System and system control method

Abstract

A system for enabling communication between a first user and a second user includes an acquisition unit and a control unit. The acquisition unit acquires real-time information of a first user. The control unit controls reflection of real-time information of a first user in a first avatar on a display device belonging to a second user and displaying a virtual space including the first avatar of the first user based on the purpose of communication.

Description

Technical Field

[0001] The present invention relates to a system for controlling the display of a user avatar and a control method of the system. Background Art

[0002] As virtual reality (VR) technology develops and becomes more widely used, it is being considered for various purposes (distribution, business, and medical, etc.) In VR, users typically use avatars (users' doubles in the system) to communicate in a virtual space.

[0003] On the other hand, there is already technology that uses a camera or a wearable device to obtain body information about a user, such as the user's facial expressions, movements, vital signs, or brain waves, etc. Therefore, it is conceivable to reflect the user's body information in the facial expressions or movements of the avatar.

[0004] Patent document 1 describes a technology that detects the movement, facial expression, or five senses of a user as a distributor, and when it is determined that the detection result satisfies a predetermined condition, changes the facial expression of an avatar to a predetermined facial expression and changes the posture of the avatar to a predetermined posture.

[0005] [Citation list]

[0006] [Patent Document]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2021-189674 Summary of the invention

[0008] Problem that the invention aims to solve

[0009] However, when it is determined that the detection result of motion, etc. satisfies the predetermined conditions, it may be inappropriate to uniformly change the facial expression and posture of the avatar. For example, in general meetings (such as business negotiations or consultations, etc.), it is not always desirable to make the avatar change the facial expression to make others clear the patient's tics or other symptoms or negative emotions.

[0010] Therefore, an object of the present invention is to provide a technology for more appropriately controlling reflection of user information in an avatar.

[0011] Solutions for solving problems

[0012] One aspect of the present invention provides a system for implementing communication between a first user and a second user, the system comprising:

[0013] an acquiring unit, configured to acquire the real-time information of the first user; and

[0014] A control unit configured to control the reflection of real-time information of the first user in the first avatar on a display device that belongs to the second user and displays a virtual space including the first avatar of the first user, based on the purpose of the communication.

[0015] An aspect of the present invention provides a system control method for implementing communication between a first user and a second user, the system control method including:

[0016] An acquisition step for acquiring real-time information of the first user; and

[0017] A control step for controlling the reflection of real-time information of the first user in the first avatar on a display device that belongs to the second user and displays a virtual space including the first avatar of the first user, based on the purpose of the communication.

[0018] Effects of the Invention

[0019] According to the present invention, the reflection of user information in the avatar can be controlled more appropriately. Description of the Drawings

[0020] Figure 1

[0021] Figure 1 is a structural diagram of a communication system according to a first embodiment.

[0022] Figure 2

[0023] Figure 2 is a structural diagram of a user terminal according to a first embodiment.

[0024] Figure 3

[0025] Figure 3 is a structural diagram of a server PC according to a first embodiment.

[0026] Figure 4

[0027] Figure 4 is a diagram showing a setting UI according to a first embodiment.

[0028] Figure 5

[0029] Figure 5 (A) to (C) of is a diagram for describing group counseling according to a first embodiment.

[0030] Figure 6A ​​​​​​​​​​​​

[0031] Figure 6A is a flowchart of a process using remote rendering according to the first embodiment.

[0032] Figure 6B

[0033] Figure 6B is a flowchart of a process using local rendering according to the first embodiment.

[0034] Figure 7

[0035] Figure 7 (A) to (C) of are diagrams showing the setting UI according to the second embodiment.

[0036] Figure 8

[0037] Figure 8 (A) to (C) of are diagrams showing the appearance of an avatar in a virtual space according to the second embodiment.

[0038] Figure 9

[0039] Figure 9 is a flowchart of a process according to the second embodiment.

[0040] Figure 10

[0041] Figure 10 is a structural diagram of a communication system according to the third embodiment.

[0042] Figure 11

[0043] Figure 11 is a flowchart of a process according to the third embodiment.

[0044] Figure 12

[0045] Figure 12 (A) and (B) of are diagrams showing the appearance of an avatar in a virtual space according to the fourth embodiment.

[0046] Figure 13

[0047] Figure 13 is a flowchart of a process according to the fourth embodiment. DETAILED DESCRIPTION

[0048] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0049] <First Embodiment>

[0050] ​​​​​​​​​​​​​​​​In the first embodiment, a communication system configured as a client - server system will be described.

[0051] Figure 1 is an overall structural diagram of the communication system according to the first embodiment. The communication system includes a server PC 101 and a plurality of user terminals 102 (terminals connected to the server PC 101 via a network such as the Internet).

[0052] Each user terminal 102 is a display device such as a PC, a smart phone, a tablet computer, or an HMD (head - mounted display). The user terminal 102 can be a controller (control device) capable of controlling such a display device. Hereinafter, the case where the user terminal 102 is an HMD will be described. Note that although the case where the HMD can be directly connected to the network will be described in the first embodiment, the HMD can be connected to the network via other devices (such as a PC or a smart phone).

[0053] Figure 2 Shows an example of the hardware structure diagram of the user terminal 102 according to the first embodiment. The user terminal 102 includes a CPU 201, a display 202, a ROM 203, a RAM 204, a network I / F 205, and an internal bus 206. The user terminal 102 includes a microphone 208, a sensor unit 209, a camera 210, a speaker 211, a storage device 212, and a near - field communication I / F 213.

[0054] The CPU 201 is a control unit that comprehensively controls various functions of the user terminal 102 via the internal bus 206 by a program stored in the ROM 203. The result of the program executed by the CPU 201 can be viewed by the user by making the display 202 display it.

[0055] The ROM 203 is a flash memory or the like. The ROM 203 stores various setting information and application programs as described above. The RAM 204 is used as a memory and a working area for the CPU 201.

[0056] The network I / F (interface) 205 is a module for connecting to the network. The microphone 208 acquires the voice emitted by the user.

[0057] The sensor unit 209 includes one or more sensors. Specifically, the sensor unit 209 includes at least any one of a GPS, a gyro sensor, an accelerometer, a proximity sensor, and a metering sensor (a sensor for measuring blood pressure, heart rate, or brain waves). In addition, the sensor unit 209 can be implemented using sensors for detecting body information (information related to the body; biometric information) to achieve authentication (such as fingerprint authentication, vein authentication, or iris authentication).

[0058] The camera 210 is a fish-eye camera (imaging unit) assembled inside the HMD. The camera 210 can acquire a captured image of the user's face. After removing the distortion caused by the fish-eye lens, the captured image is stored in the RAM 204.

[0059] The speaker 211 reproduces the voice of the user participating in the communication system, sound effects, BGM, etc.

[0060] The storage device 212 is a storage medium. In addition, the storage device 212 is a device for storing various data such as applications.

[0061] The near-field communication I / F 213 is an interface used in communication with the user's controller. The user can input gestures to the user terminal 102 by moving the controller that the user is holding. In addition, the user can issue commands to the user terminal 102 by operating buttons or joysticks provided in the controller, etc. The controller may include sensors for measuring the user's heart rate, pulse, sweating, etc. In addition, the near-field communication I / F 213 can communicate with a wearable device worn by the user to obtain the user's heart rate, pulse, sweating, etc. Furthermore, the near-field communication I / F 213 can communicate with devices (such as cameras or sensor groups) installed inside the room where the user is present.

[0062] Figure 3 An example of a hardware configuration diagram of the server PC 101 according to the first embodiment is shown.

[0063] As Figure 3 shown, the server PC 101 includes a display unit 301, VRAM 302, BMU 303, keyboard 304, PD305, CPU 306, storage device 307, RAM 308, ROM 309, and flexible disk 310. The server PC 101 includes a microphone 311, speaker 312, network I / F 313, and bus 314.

[0064] For example, the display unit 301 displays a live view image, icon, message, menu, or other user interface information.

[0065] A moving image to be displayed on the display unit 301 is drawn in the VRAM 302. The data of the moving image generated in the VRAM 302 is transmitted to the display unit 301 according to a specified rule, and thus is displayed on the display unit 301.

[0066] The BMU (bit movement unit) 303 controls data transfer between, for example, multiple memories (such as VRAM 302 and other memories). In addition, the BMU (bit movement unit) 303 controls data transfer between, for example, a memory and each I / O device (such as the network I / F 313).

[0067] The keyboard 304 includes various keys for user input of characters and the like.

[0068] For example, the PD (pointing device) 305 is used to issue instructions for content (icons, menus, etc.) or for dragging and dropping an object displayed on the display unit 301.

[0069] The CPU 306 is a control unit for controlling each component based on the OS and programs (control programs) stored in the storage device 307, ROM 309, or flexible disk 310.

[0070] The storage device 307 is an HDD (hard disk drive) or an SSD (solid state drive). The storage device 307 stores various control programs and various data to be temporarily stored, etc.

[0071] The RAM 308 includes a working area for the CPU 306, an area for storing data during error handling, and an area for loading control programs, etc.

[0072] The ROM 309 stores various control programs used in the server PC 101 and data to be temporarily stored, etc.

[0073] The flexible disk 310 stores various control programs and various data (data that needs to be temporarily stored), etc.

[0074] The microphone 311 acquires the sound around the server PC 101. The speaker 312 outputs the sound included in the data of the moving image.

[0075] The network I / F 313 communicates with the user terminal 102 via the network. The bus 314 includes an address bus, a data bus, and a control bus.

[0076] The control program can be provided to the CPU 306 from the storage device 307, ROM 309, or flexible disk 310, or can be provided to the CPU 306 via the network from other information processing devices, etc. through the network I / F 313.

[0077] Figure 4 FIG. is a diagram showing a setting UI (user interface) 401 of the communication system according to the first embodiment. The setting UI 401 is displayed on the display 202 of the user terminal 102. The user uses the setting UI 401 to set how to reflect user body information (information related to the body; biometric information) including user symptoms in the user avatar and transmit it to other users (reflection setting). In Figure 4Among them, five settings are displayed, and each setting has a priority from 1 to 5. Additionally, in the setting UI 401, the UI areas 402 to 404 each indicate a condition, and the UI area 405 indicates the processing when the condition is met.

[0078] Hereinafter, the user for whom the configuration reflects the settings will be referred to as the "first user", and other users (users other than the first user and participating in the same virtual space group as the first user) will be referred to as the "other user". Additionally, the avatar of the first user will be referred to as the "used avatar".

[0079] The UI area 402 is a UI area for setting the communication purpose (e.g., conducting a consultation or participating in a business negotiation).

[0080] The UI area 403 is a UI area for setting the role of the other user (e.g., a counselor or a patient). Note that the case where there is no input content in the UI area 403 indicates that the role of the other user can be any role.

[0081] The UI area 404 is a UI area for setting the type of body information (body information type) to be reflected in the avatar. Examples of body information types include tics, smiles, and nervousness. A tic refers to a sudden body movement or vocalization that occurs unexpectedly.

[0082] The UI area 405 is a UI area for setting the degree of reflection (reflection method) of the body information indicated by the body information type to the avatar. When reflecting the user's facial expression, etc. in the avatar, the user can select one of the following options: "reflect as it is", "emphasize reflection", "suppress reflection", and "do not reflect".

[0083] For example, assume a case where a group consultation for tic symptoms is conducted in a virtual space. "Group consultation" is a consultation technique that enables multiple patients to be together. In this case, multiple users with different roles (identities) of counselors and patients participate in a group (group) in the virtual space. The first user, who is a patient, may be aware of his or her tic symptoms and may not want other patients to see the tic symptoms. In this case, for example, the first user adopts the setting shown in the setting group 406 such that "when the communication purpose is to conduct a consultation, the tic is not reflected in the used avatar to be shown to the other user whose role is the patient role" (adopts the setting with a priority of 2). Additionally, the first user adopts the setting such that "when the communication purpose is to conduct a consultation, the tic is reflected in the used avatar to be shown to the other user whose role is the counselor role" (adopts the setting with a priority of 1).

[0084] In other words, when reflecting his / her symptoms in the usage avatar, the first user can set whether to show the symptoms (and the degree to show the symptoms) according to the role of the other user whose usage avatar is to be shown.

[0085] In addition, for example, assume the following situation: When participating in a business negotiation in a virtual space, the first user hopes that his / her usage avatar reflects a friendly facial expression (such as a smile, etc.) in an emphasized manner and reflects nervousness in a suppressed manner in order to relax the other party's guard. In this case, the first user can adopt the setting shown in setting group 407 such that "when the communication purpose is to participate in a business negotiation, the usage avatar to be shown to the other user reflects the first user's smile in an emphasized manner and reflects the first user's nervousness in a suppressed manner".

[0086] Note that in the setting UI 401, for each communication purpose, the type of body information and the degree of reflection to the avatar can be preset.

[0087] In addition, among the multiple settings input in the setting UI 401, the communication system preferably uses the setting with a smaller assigned priority number. For example, assume the following situation: In the Figure 4 shown example, the communication purpose is to conduct a consultation, the role of the other user is a counselor role, and the reflection of tics in the usage avatar is to be controlled. In this case, although the settings with priorities 1 and 5 can be used, the communication system uses the setting with a lower priority (in other words, the setting with priority 1). Therefore, the communication system exerts control such that the tic symptoms occurring in the first user are reflected "as they are" in the usage avatar.

[0088] In addition, the communication system can make the usage avatar reflect not only visible information such as the facial expression or movement of the first user, but also other information. For example, when performing remote medical treatment or consultation in a virtual space, the user terminal 102 uses the sensor unit 209 to obtain the patient's body temperature or heart rate, or uses the camera 210 to obtain eye movement. In addition, the user terminal 102 can estimate the degree of tension or calmness of the first user based on the obtained information and reflect the estimation result in the avatar.

[0089] Although the above setting UI is intended to be used before the first user joins a group in the virtual space, there can also be a UI set while the first user is watching the virtual space. Such a UI will be described later with reference to Figure 5 in (C).

[0090] Figure 5(A) to (C) are diagrams showing group consultations in a virtual space. In this case, it is assumed that patient A has adopted settings such that "when the communication purpose is consultation, tics are not reflected in the avatar to be shown to the other user with the role of patient". In addition, it is assumed that patient A has adopted settings such that "when the communication purpose is consultation, tics are reflected in the avatar to be shown to the other user with the role of counselor". In addition, it is assumed that four participants (the main counselor, the deputy counselor, patient A, and patient B) are participating in the group consultation. Note that avatar 501 is the avatar of the main counselor, and avatar 502 is the avatar of patient A.

[0091] Figure 5 (A) shows the virtual space displayed on the user terminal 102 of the deputy counselor, and Figure 5 (B) shows the virtual space displayed on the user terminal 102 of patient B. Note that the virtual space displayed on the user terminal 102 of the deputy counselor represents the space (field of view) visible from the avatar of the deputy counselor. Note that the virtual space displayed on the user terminal 102 of patient B represents the space visible from the avatar of patient B.

[0092] In this case, the avatars of the deputy counselor and patient B are located at different positions in the virtual space. Therefore, the range of the displayed avatars (avatar 501 of the main counselor and avatar 502 of patient A) is different between the user terminal 102 of the deputy counselor and the user terminal 102 of patient B. It is assumed that the avatar of patient B is not included in the display of the user terminal 102 of the deputy counselor. It is assumed that the avatar of the deputy counselor is not included in the display of the user terminal 102 of patient B.

[0093] Now, assume a situation where patient A has a motor tic of frowning during the consultation. Since information related to motor tics is necessary for the counselor, as Figure 5 shown in (A), in the virtual space being viewed by the deputy counselor, avatar 502 of patient A is frowning (the motor tic of patient A is reflected in avatar 502 of patient A). In a similar manner, in the virtual space being viewed by the main counselor, avatar 502 of patient A is also frowning.

[0094] On the other hand, as Figure 5 shown in (B), in the virtual space being viewed by patient B, the facial expression of avatar 502 of patient A does not change. This is because patient A has adopted the setting such that "tics are not reflected in the avatar to be shown to the other user with the role of patient" as shown in setting group 406.

[0095] Figure 5Example (C) is another example of the UI that the first user uses to set "how to show the use of the avatar to the other user in the virtual space". For example, Patient B uses a controller or the like to select the avatar 501 of the main counselor, and then issues an instruction on how to show his / her avatar. Accordingly, the setting screen 503 is displayed in the virtual space. Patient B uses the setting screen 503 to set how he / she will present his / her avatar to the main counselor. Accordingly, even when the virtual space is being displayed, the first user can easily change the setting.

[0096] Figure 6A and Figure 6B The flowchart in shows the processing of the communication system according to the first embodiment. Figure 6A The flowchart in shows the processing of a method (a method called remote rendering) of rendering the video displayed on each user terminal 102 using the server PC 101. On the other hand, Figure 6B shows the processing of a method (a method called local rendering) of rendering the video using the user terminal 102. The communication system according to the first embodiment can execute either of these two processes.

[0097] Reference will be made to Figure 6A the flowchart shown to describe the processing using remote rendering.

[0098] Steps S601 to S603 represent the processing in which the first user sets how to reflect the first user's body information in his / her use of the avatar. Note that this processing is performed between the user terminals 102 of all users who join the group in the virtual space and the server PC 101. Hereinafter, the user terminal 102 of the first user will be referred to as "user terminal 102A", and each component of the user terminal 102A will have the suffix "A". For example, the display 202 of the user terminal 102A will be referred to as "display 202A", and the CPU 201 of the user terminal 102A will be referred to as "CPU 201A".

[0099] In step S601, the CPU 201 (CPU 201A) of the user terminal 102A receives from the first user the reflection setting (the setting related to how to reflect the first user's body information in the use of the avatar and deliver it to the other user). Specifically, the CPU 201A obtains the setting input by the user in the Figure 4 shown setting UI (the setting that correlates the communication purpose, the role of the other user, the type of body information, and the degree of reflection) as the reflection setting.

[0100] In step S602, the CPU 201A sends the received reflection setting to the server PC 101.

[0101] In step S603, the CPU 306 of the server PC 101 records (stores) the received reflection settings in the storage device 307 or the like.

[0102] The subsequent steps S604 to S606 represent the process for the first user to join a group in the virtual space. Although omitted in Figure 6A , this process is executed between the user terminal 102 of all users joining the group in the virtual space and the server PC 101.

[0103] In step S604, the CPU 201A receives from the first user an instruction to participate in a group in the virtual space. At this time, the CPU 201A acquires the identification information of the virtual space of the group that the first user desires to join (hereinafter referred to as the "desired space").

[0104] In step S605, the CPU 201A sends the identification information of the desired space to the server PC 101 to request to participate in the group in the desired space.

[0105] In step S606, the CPU 306 makes the first user join the group in the desired space corresponding to the acquired identification information.

[0106] The subsequent steps S607 to S616 represent the process to be repeatedly executed (loop process). This loop process is repeated until all users including the first user leave the group in the desired space.

[0107] In step S607, the CPU 201A acquires the real-time (current) body information of the first user. For example, the CPU 201A acquires at least any one of the following body information: voice, emotion, facial expression, blood pressure, heart rate, stress level, body temperature, sweating amount, brain wave, pulse rate, posture, and movement (including eye movement). For example, the CPU 201A can acquire a captured image of the first user as body information.

[0108] For example, the CPU 201A controls the microphone 208A to acquire the voice emitted by the first user. In this case, the CPU 201A can also use existing technologies for analyzing voice emotion to estimate the emotion of the first user based on the acquired voice.

[0109] The CPU 201A can control the camera 210A to capture an image of the user's face. In this case, the CPU 201A can use facial expression analysis technology to analyze (acquire) the facial expression of the first user based on the captured image. In addition, the CPU 201A can analyze the eye movement of the first user based on the captured image to estimate the psychology of the first user. Furthermore, the CPU 201A can use vital sign data analysis technology to estimate the blood pressure, heart rate, and / or stress level of the first user.

[0110] In addition, the CPU 201A can control the sensor unit 209A and measure at least any one of the blood pressure, heart rate, body temperature, sweating amount, brain waves, etc. of the first user.

[0111] The CPU 201A can use the near - field communication I / F 213 to communicate with the controller held by the first user or the wearable device worn by the first user. The CPU 201A can acquire the information obtained by the controller or the wearable device (any one of the heart rate, body temperature, sweating amount, brain waves, etc. of the first user). The CPU 201A can use the near - field communication I / F 213 to communicate with the camera installed inside a room or the like to acquire the captured image of the first user taken by the camera. In addition, the CPU 201A can acquire the posture information or movement operation of the first user based on the captured image.

[0112] The CPU 201A can use the near - field communication I / F 213 to connect to the sensor group installed inside the room where the first user is present to acquire the vital sign information of the user.

[0113] In step S608, the CPU 201A sends the body information of the first user acquired in step S607 to the server PC 101.

[0114] Note that in step S607, when the CPU 201A determines that the first user with tic symptoms has frowned, the CPU 201A can determine that a motor tic has occurred in the first user. In addition, the CPU 201A can determine whether the first user with dizziness symptoms is experiencing dizziness based on the eye movement of the first user. In other words, the CPU 201A can refer to the information indicating the disease of the first user (disease information) to determine whether symptoms have appeared in the first user. Furthermore, the CPU 201A can control how to reflect body information such as motor tics or dizziness in the use of the avatar based on the judgment result related to whether symptoms have appeared in the first user.

[0115] For example, when the CPU 201A determines that the disease indicated by the disease information of the first user has occurred in the first user, in step S608, the CPU 201A may send body information related to the symptoms of the first user (for example, information related to the frowning behavior). On the other hand, when the CPU 201A determines that the disease indicated by the disease information of the first user has not occurred in the first user, in step S608, the CPU 201A may not send body information related to the symptoms of the first user. Therefore, when it is determined that the symptoms indicated by the disease information of the first user have occurred in the first user, in step S612, which will be described later, the body information related to the symptoms can be arranged to be reflected in the use of the avatar. On the other hand, when it is determined that the symptoms indicated by the disease information of the first user have not occurred in the first user, in step S612, which will be described later, the body information related to the symptoms can be arranged not to be reflected in the use of the avatar. In other words, even when the same facial expression of frowning is presented, it is possible to change whether to convey the facial expression to the other user by reflecting the facial expression in the use of the avatar according to the disease of the first user.

[0116] In step S609, the CPU 306 of the server PC 101 receives the body information of the first user.

[0117] In step S610, the CPU 306 obtains (judges) the purpose of the communication carried out in the desired space.

[0118] For example, the CPU 306 uses the UI displayed on the user terminal 102 to obtain the communication purpose input by any user (any user participating in the group in the desired space).

[0119] In addition, when the communication purpose is linked to the information forming the desired space (for example, when the information related to the "virtual space for consultation" is linked to the information forming the desired space), the CPU 306 can judge the purpose from this information. The CPU 306 can estimate (judge) the communication purpose based on the information related to the account of at least any one of the multiple users participating in the group in the desired space. For example, when a user with a counselor's account is participating in the group in the desired space, the CPU 306 can estimate that the communication purpose is "conduct a consultation".

[0120] The CPU 306 can analyze the appearance of each avatar in the desired space to estimate the communication purpose. For example, when an avatar wearing a white coat exists in the desired space, the CPU 306 can estimate that the communication purpose is "conduct a medical examination or consultation".

[0121] The processing of subsequent steps S611 to S615 is a loop process, which is repeated the following number of times, and this number is the same as the number of users (except the first user) who participate in the group and watch the video of the desired space (virtual space). Hereinafter, one of the users who watch the video of the desired space will be referred to as the "second user". Hereinafter, the user terminal 102 of the second user will be referred to as "user terminal 102B", and each component of the user terminal 102B will have the suffix "B". For example, the display 202 of the user terminal 102B will be referred to as "display 202B", and the CPU 201 of the user terminal 102B will be referred to as "CPU 201B".

[0122] In step S611, the CPU 306 determines (confirms) the role (identity) of the second user.

[0123] For example, the CPU 306 obtains information about the role of the second user based on information related to the account of the second user. For example, when the communication purpose is to conduct a consultation, the CPU 306 determines whether the role of the second user is a consultant role based on information related to the accounts of each user managed by the communication system.

[0124] The CPU 306 can determine the role of the second user based on information obtained from an external system. For example, the CPU 306 queries the electronic medical record system in the hospital to determine whether the role of the second user is a consultant role (whether the second user is registered as a consultant) based on the query result.

[0125] For example, when the first user can set the role of the other user who participates in the group in the desired space, the CPU 306 can refer to the setting information to determine the role of the second user. Using this method, for example, the first user as a patient can classify the second user as a "trusted consultant" or an "untrusted consultant". In this case, the CPU 306 can exert control so that the tic symptoms of the first user are reflected in the used avatar that the "trusted consultant" is to view, but not reflected in the used avatar that the "untrusted consultant" is to view.

[0126] In step S612, the CPU 306 controls the used avatar of the first user based on the physical information of the first user, the communication purpose, and the role of the second user.

[0127] The case where the first user exhibits motor tics (motor tics involving frowning) will be used as an example to describe the processing of step S612.

[0128] For example, assume a case where in step S607, the CPU 201A controls the camera 210 to acquire a captured image of the face of the first user. In this case, in step S607, the CPU 201A analyzes the facial expression from the captured image to obtain information regarding whether the first user has frowned. In addition, considering that the user has a tic disorder, the CPU 201A acquires information indicating that a motor tic has occurred as the body information of the first user. Further, in step S608, the CPU 201A transmits the body information of the first user to the server PC 101 via the network I / F 205.

[0129] As a result, in step S612, the CPU 306 refers to the reflection settings of the first user recorded in step S603. Now, as Figure 4 shown, assume that the settings of the first user involve "in counseling, reflect tics in the avatar to be viewed by the user acting as the counselor, but do not reflect tics in the avatar to be viewed by the user acting as the patient". In addition, information indicating that a motor tic has occurred has been acquired as the body information of the first user. Therefore, according to the reflection settings, when the communication purpose is counseling and the role of the second user is the counselor role, the CPU 306 controls the avatar of the first user to frown. On the other hand, when the communication purpose is not counseling or the role of the second user is the patient role, the CPU 306 controls the avatar of the first user not to frown.

[0130] In step S613, the CPU 306 generates a 3D scene of the desired space including the avatar of the first user controlled in step S612. For example, the CPU 306 generates a 3D scene in a data format capable of describing three-dimensional computer graphics (such as X3D, etc.).

[0131] In step S614, the CPU 306 renders the 3D scene of the desired space and generates a video of the desired space as viewed from the avatar of the second user (the viewpoint of the avatar). In this case, the CPU 306 generates a video in a data format such as MP4.

[0132] In step S615, the CPU 306 sends the video generated in step S614 to the user terminal 102B.

[0133] In step S616, the CPU 201B receives the video. The CPU 201B displays the video on the display 202B.

[0134] Next, the processing using local rendering will be described with reference to the Figure 6B flowchart shown. When performing the processing using local rendering (when performing the Figure 6B processing shown), Figure 6AThe shown steps S614 and S615 are replaced by steps S631 and S632. On the other hand, the processing in the other steps (steps S601 to S613 and S616) when using local rendering is performed in a similar manner to the case when using remote rendering ( Figure 6A the shown case). Therefore, hereinafter, only steps S631 and S632 will be described.

[0135] In step S631, the CPU 306 sends the 3D scene generated in step S613 to the user terminal 102B.

[0136] In step S632, the CPU 201B renders the 3D scene of the desired space received, and then generates frames of a video of the desired space as viewed from the avatar of the second user.

[0137] As described above, since the communication system operates according to the reflection settings described in the reference Figure 4 for example, a situation where only the avatar of patient A frowns in the virtual space observed by the counselor can be realized. In other words, the use of the avatar can be more appropriately controlled based on the role of the other user and the communication purpose.

[0138] Note that in the above description, for simplicity, in the case where the second user is the main counselor and in the case where the second user is the assistant counselor, the processing for generating the 3D scene in the virtual space in steps S612 to S613 is performed separately. However, in reality, if the 3D scene in the virtual space generated in steps S611 to S612 is repeatedly used for multiple other users with the same role, the processing will be more efficient.

[0139] <Second Embodiment>

[0140] In the first embodiment, an example is shown in which the degree of reflection of body information in the use of the avatar is set by having the first user configure a reflection setting that correlates the communication purpose and the degree of reflection of body information with each other. However, depending on the communication purpose, a user in a specific role (identity) may want to determine the degree to which the body information of other users is to be reflected. For example, in group counseling, when there are patients with severe tic symptoms and patients with mild tic symptoms, the counselor may want to make the symptoms of the patients with mild tic symptoms more prominent. Considering this, in the second embodiment, a communication system that enables a given user to set the degree to which body information is reflected in the avatars of other users will be described.

[0141] In a manner similar to the first embodiment, it is assumed that four participants (the main counselor, the assistant counselor, patient A, and patient B) are participating in group counseling (a group) in a virtual space. It is also assumed that patient A is a severe patient with severe motor tic reactions, and patient B is a mild patient with mild motor tic reactions. Note that in the second embodiment, when the user's motor tic reaction is severe (according to the severity of the motor tic reaction), the communication system (CPU 306) increases the size of the user's avatar.

[0142] Figure 7 (A) to (C) of FIG. are diagrams showing examples of the setting UI according to the second embodiment. The setting UI 701 is displayed on the user terminal 102 of the counselor. The counselor uses the setting UI 701 to set how the physical information of the patient (including the patient's symptoms) is to be displayed on the user terminal 102 of the counselor (configuring the reflection setting).

[0143] The UI area 702 is an area representing the type of physical information. In Figure 7 (A) to (C) of FIG., motor tics and vocal tics are selected as the types of physical information.

[0144] The UI area 703 is a UI area for setting the degree of reflection of the physical information on the avatar. In Figure 7 (A) to (C) of FIG., as an example, a slider 704 is displayed in the UI area 703.

[0145] The slider 704 is a UI area for setting whether to "suppress" or "emphasize" the user's tic reaction when the avatar is to reflect the user's tic reaction. Moving the pointer 706 on the slider 704 to the left "suppresses" the user's tic reaction. Moving the pointer 706 to the right "emphasizes" the user's tic reaction.

[0146] Figure 7 (A) of FIG. shows an example where the degree of reflection of patient A's motor tic reaction in the avatar is set to neither "suppress" nor "emphasize". Figure 7 (B) of FIG. shows an example where the degree of reflection of patient A's motor tic reaction in the avatar is set to "suppress". Figure 7 (C) of FIG. shows an example where the degree of reflection of patient B's motor tic reaction in the avatar is set to "emphasize".

[0147] Note that instead of the counselor using the setting UI 701 as described above to set the degree of reflection, the CPU 306 can automatically set the degree of reflection according to the patient's disease information. For example, the milder the symptoms indicated by the patient's disease information, the greater the degree to which the CPU 306 reflects the physical information related to the disease information in the avatar of that patient. Therefore, the time and effort required for the user to set the degree of reflection can be reduced.

[0148] Figure 8 (A) to (C) thereof are diagrams showing the avatars of Patient A and Patient B as displayed on the user terminal 102 of, for example, the main counselor. Avatar 801 is the avatar of Patient A, and avatar 802 is the avatar of Patient B.

[0149] Figure 8 (A) thereof shows the two avatars displayed when neither Patient A nor Patient B is experiencing motor tics.

[0150] Figure 8 (B) thereof shows Figure 7 the two avatars when motor tics occur simultaneously in Patient A and Patient B in a case where the degree of reflection of the reaction shown in (A) thereof to the avatars of Patient A and Patient B is set to neither "suppression" nor "emphasis". In this case, Figure 8 (B) thereof shows that the motor tic reaction of Patient A is severe and the motor tic reaction of Patient B is mild.

[0151] Figure 8 (C) thereof shows the two avatars in a state where the degree of reflection of the reaction to the avatar in the UI of Patient A in (B) thereof is set to "suppression" and the degree of reflection of the reaction to the avatar in the UI of Patient B in (C) thereof is set to "emphasis". Compared with the case shown in Figure 7 (B) thereof, the two avatars are shown in a state where the size of the avatar of Patient A is smaller and the size of the avatar of Patient B is larger. Therefore, a display that makes the occurrence of motor tics in Patient A and Patient B more prominent is achieved. Figure 7 (C) thereof, the two avatars are shown in a state where the size of the avatar of Patient A is smaller and the size of the avatar of Patient B is larger. Therefore, a display that makes the occurrence of motor tics in Patient A and Patient B more prominent is achieved. Figure 8 (B) thereof, the two avatars are shown in a state where the size of the avatar of Patient A is smaller and the size of the avatar of Patient B is larger. Therefore, a display that makes the occurrence of motor tics in Patient A and Patient B more prominent is achieved.

[0152] Next, the processing according to the second embodiment will be described with reference to the flowchart in Figure 9 by using Figure 9 . In the flowchart shown in Figure 9 , steps S601 and S602 in the flowchart shown in Figure 6A are replaced by steps S901 and S902, and the steps after step S603 are the same as the steps in the flowchart shown in Figure 6A . Therefore, only steps S901 and S902 will be described.

[0153] In step S901, the CPU 201B of the user terminal 102B receives a reflection setting for instructing the degree of reflection of the body information (including symptoms) of the first user in the avatar of the first user. In this case, when the second user uses Figure 7When the UI shown in (A) sets the degree of reflection of the first user's body information in the first user's avatar to "emphasis" or "suppression", the CPU 201B accepts this setting as the reflection setting.

[0154] In step S902, the CPU 201B sends information related to the reflection setting to the server PC 101.

[0155] Based on the above, the user who actually views the avatar can set the degree of reflection of the body information in the avatars of other users. Additionally, when setting the avatar to reflect the movements or voices of other users, it is also possible to set for each user which movement or voice to emphasize or suppress and the degree of emphasis or suppression.

[0156] <Third Embodiment>

[0157] In the first and second embodiments, a communication system as a client - server system for connecting the server PC 101 and multiple user terminals 102 is described. However, the communication system can also be implemented by a system that does not include the server PC 101. Considering this, in the third embodiment, the case of constructing the communication system described in the first embodiment using a system that does not include the server PC 101 will be described. Note that the communication system described in the second embodiment can also be constructed using a system that does not include the server PC 101.

[0158] Figure 10 is a system configuration diagram of the communication system according to the third embodiment. The communication system includes multiple user terminals 102 connected by P2P (peer - to - peer) through a network such as the Internet. Since Figure 10 each of the multiple user terminals 102 shown shares the same structure as the user terminal 102 according to the first embodiment, no detailed description will be provided. Additionally, the setting UI according to the third embodiment is the same as the setting UI 401 according to the first embodiment described with reference to Figure 4

[0159] Figure 11 is a flowchart showing the processing of the communication system according to the third embodiment.

[0160] In step S1101, the CPU 201A of the user terminal 102A of the first user accepts the reflection setting from the first user in a manner similar to step S401.

[0161] The subsequent steps S1102 to S1106 represent the process of the first user joining a group in the desired space (virtual space). Although this process is Figure 11 omitted here, this process is executed among the user terminals 102 of all other users who join the group in the desired space. ​

[0162] In step S1102, the CPU 201A receives a participation instruction from the first user to join a group in the desired space and obtains the identification information of the desired space in a manner similar to step S604.

[0163] In step S1103, the CPU 201A sends the identification information of the desired space to the user terminal 102B (the user terminal 102 of the other user) in a manner similar to step S605. Therefore, the CPU 201A notifies that the first user will join the group in the desired space.

[0164] In step S1104, the CPU 201B records that the first user has joined the group in the desired space.

[0165] In step S1105, the CPU 201B sends information related to the second user to the user terminal 102A. The information related to the second user includes information related to the role of the second user. The acquisition of the role of the second user can be achieved by a method similar to step S611.

[0166] In step S1106, the CPU 201A receives the information related to the second user.

[0167] In step S1107, the CPU 201A obtains the purpose of the communication to be performed in the desired space in a manner similar to step S610.

[0168] The processing of subsequent steps S1110 to S1117 is a loop process, and this loop process is repeated until all users including the first user leave the virtual space.

[0169] In step S1110, the CPU 201A obtains the real-time (current) body information of the first user in a manner similar to step S607.

[0170] The processing of steps S1112 to S1113 is repeated the same number of times as the number of second users (the number of user terminals 102B communicating with the user terminal 102A) other than the first user participating in the group in the desired space.

[0171] In step S1112, the CPU 201A controls the use avatar of the first user based on the body information, communication purpose, and information related to the second user in a manner similar to step S612.

[0172] In step S1113, the CPU 201A generates a 3D model of the use avatar controlled in step S1112 and sends the generated 3D model to the user terminal 102B.

[0173] In step S1114, the CPU 201B receives the 3D model of the avatar of the first user.

[0174] In step S1115, the CPU 201B generates a 3D scene of the desired space including the avatar of the first user. The CPU 201B generates the 3D scene in a data format capable of describing three-dimensional computer graphics (such as X3D, etc.).

[0175] In step S1116, the CPU 201B renders the 3D scene of the desired space generated in step S1115 and generates a frame of the video of the desired space as viewed from the viewpoint of the second user.

[0176] In step S1117, the CPU 201B displays the generated video of the desired space on the display 202B.

[0177] As described above, according to the third embodiment, even in a communication system that does not include the server PC 101 (server), it is possible to more appropriately control the reflection of body information in the avatar.

[0178] <Fourth Embodiment>

[0179] In the fourth embodiment, the communication system changes (updates) the information required to control the avatar according to the virtual space of the group to which the user belongs.

[0180] In the fourth embodiment, in a manner similar to the first embodiment, the communication system is configured as described with reference to Figures 1 to 3 As described. In the fourth embodiment, the communication system estimates the user's emotion based on the acquired body information and reflects the estimated emotion in the facial expression of the user's avatar. Note that the "body information" according to the fourth embodiment is information other than emotion.

[0181] Figure 12 (A) and (B) of are diagrams showing the appearance of the avatars of the participants participating in the communication system. As an example, assume a situation where multiple participants use the virtual space to participate in a business negotiation. Negotiation opponent A, negotiation opponent B, and user C as the presenter are participating in the business negotiation. The ranges of the virtual spaces displayed on the user terminals 102 of negotiation opponent A, negotiation opponent B, and user C are different from each other. For example, Figure 12 (A) and (B) of show the virtual space displayed on the user terminal 102 of negotiation opponent B. In this case, the avatar 1201 is the avatar of negotiation opponent A, and the avatar 1202 is the avatar of user C.

[0182] Figure 12(A) and (B) are shown in the following example, in which, due to the estimation of emotions based on body information, it is estimated that user C, who is the presenter, is feeling anxious and tense.

[0183] Figure 12 (A) is a diagram depicting the display of the user terminal 102 when the process for changing body information according to the fourth embodiment is not performed (when the acquired body information is used as it is for emotion estimation). In the virtual space displayed on the user terminal 102 of negotiation partner B, as Figure 12 shown in (A), since user C is experiencing anxious and tense emotions in the real space, the emotions are also reflected in the facial expression of the avatar 1202.

[0184] On the other hand, Figure 12 (B) is a diagram depicting the display of the user terminal 102 when the process for changing body information according to the fourth embodiment is performed (when the acquired body information is changed and the changed body information is subsequently used for emotion estimation). In the virtual space displayed on the user terminal 102 of negotiation partner B, the anxiety and tension of user C are not reflected in the facial expression of user C's avatar 1202 (the anxiety and tension are suppressed).

[0185] Figure 13 is a flowchart showing the processing of the communication system according to the fourth embodiment. Figure 13 The processing represented by the flowchart in Figure 6A is a process that uses the remote rendering method in a similar manner to Figure 6B Note that the communication system according to the fourth embodiment can also be implemented using the

[0186] The processing represented by the flowchart in Figure 13 is implemented by the CPU 201 controlling each unit of the user terminal 102 according to the program stored in the ROM 203.

[0187] Steps S1301 to S1303 represent the processing for the first user to join a group in the virtual space. This processing is executed between the user terminals 102 of all users joining the virtual space and the server PC 101.

[0188] In step S1301, the CPU 201A receives a participation instruction from the first user to join the virtual space. In doing so, the CPU 201A acquires the identification information of the virtual space (desired space) that the first user desires to join.

[0189] In step S1302, the CPU 201A sends the identification information of the desired space to the server PC 101 to request the server PC 101 to participate in a group in the virtual space.

[0190] In step S1303, the CPU 306 of the server PC 101 makes the first user join the group in the desired space corresponding to the identification information.

[0191] The processing of subsequent steps S1305 to S1315 is a loop process, and this loop process is repeatedly executed until all users including the first user leave the group in the virtual space.

[0192] In step S1305, the CPU 306 sends the information related to the accounts of all users participating in the group in the desired space to the user terminal 102A.

[0193] In step S1306, the CPU 201A determines the communication purpose. For example, the CPU 201A determines the communication purpose based on the information related to the tool used by the first user in the desired space. For example, when the first user is using a presenter tool, the CPU 201A may determine that the communication purpose is to hold a meeting (presentation).

[0194] In addition, in step S1306, the CPU 201A also acquires (determines) the information (related information) related to the communication. Note that the related information can be understood as being included in the communication purpose.

[0195] (1) For example, the CPU 201A determines the role (identity) of the first user based on the information related to the tool used by the first user in the desired space. When the first user is using a presenter tool, the CPU 201A may determine that the first user is a presenter.

[0196] (2) For example, when the communication purpose is to hold a meeting, the CPU 201A determines the type of the meeting (the relationship between multiple users participating in the meeting) based on the accounts of the users participating in the group in the desired space. For example, the CPU 201A determines whether the type of the meeting to be held is a meeting with business partners, an internal company meeting, or a meeting with friends. The identification information of the user accounts registered in the user terminal 102 or the server PC 101 (virtual space) is used to determine the type of the meeting. Note that in step S1308 to be described later, the CPU 201A changes the body information of the first user acquired in step S1307 according to the type of the meeting (the relationship between multiple users participating in the meeting).

[0197] (3)For example, the CPU 201A determines the nationality of the participant based on the accounts of the users who are part of the group in the desired space. The identification information of the user accounts registered in the user terminal 102 or the server PC 101 (virtual space) is used for this determination. Note that in step S1308, which will be described later, the CPU 201A changes the body information of the first user obtained in step S1307 according to the nationality of the participant.

[0198] (4)For example, the CPU 201A determines the content of the event based on the information related to the event to be held in the desired space (on the platform in the desired space). Note that when the content of the event is to hold a demonstration meeting or participate in a business negotiation, in step S1308, which will be described later, the CPU 201A changes the body information of the first user obtained in step S1037.

[0199] In step S1307, the CPU 201A obtains the body information of the first user.

[0200] In addition, the CPU 201A determines the emotional information of the first user based on the obtained body information (emotion determination). For example, the correspondence between the output result of the body information and the emotion is obtained in advance in a verifiable manner, and a table showing this correspondence is stored in the ROM 203 or the server PC 101. The CPU 201A determines whether the obtained body information matches the specific emotion pattern described in the table to determine the emotional information of the first user.

[0201] In step S1308, the CPU 201A controls the body information of the first user obtained in step S1307 based on the communication purpose and related information determined in step S1306.

[0202] (1)For example, assume a case where the CPU 201A determines that the communication purpose is to hold a meeting and the first user is the presenter (the CPU 201A determines that the communication purpose is to hold a meeting with the first user as the presenter). In this case, the CPU 201A changes the body information of the first user so that "the tense facial expressions or gestures in the emotion determination are suppressed". For example, when information related to blood pressure or heart rate is obtained as the body information, the CPU 201A reduces the blood pressure or heart rate by a specified value.

[0203] (2)For example, assume a case where the CPU 201A determines that the communication purpose is to hold a meeting and the type of the meeting is a meeting with a business partner (the communication purpose is to hold a meeting with a business partner). In this case, the CPU 201A changes the body information of the first user so that "the tense facial expressions or gestures in the emotion determination are suppressed".

[0204] (3) For example, assume a case where the CPU 201A determines that the communication purpose is to hold a meeting and the type of the meeting is a meeting with friends (the communication purpose is to hold a meeting with friends). In this case, the CPU 201A does not change the body information of the first user.

[0205] (4) For example, assume a case where the CPU 201A determines that the communication purpose is to hold a meeting and users of multiple nationalities are to participate in the meeting (the communication purpose is to hold a meeting in which users of multiple nationalities are to participate). In this case, the CPU 201A changes the body information of the first user such that "a tense facial expression or gesture in emotion judgment is suppressed and a smile is emphasized". For example, when obtaining a stress level as body information, the CPU 201A reduces the stress level by a prescribed value such that a smile is emphasized in emotion judgment. Alternatively, when obtaining information related to movement as body information, the CPU 201A emphasizes the information of the mouth-opening movement such that a smile is emphasized in emotion judgment.

[0206] (5) For example, assume a case where the CPU 201A determines that the communication purpose is to hold a meeting and the content of the meeting is to conduct a lecture meeting or participate in a business negotiation (the communication purpose is to conduct a lecture meeting or participate in a business negotiation). In this case, the CPU 201A changes the body information of the first user such that "a tense facial expression or gesture in emotion judgment is suppressed".

[0207] (6) For example, assume a case where the CPU 201A determines that the communication purpose is to play a game and the type of the game is a game that requires a poker face (the communication purpose is to play a specific game). In this case, the CPU 201A changes the body information of the first user such that "a tense facial expression or gesture in emotion judgment is suppressed".

[0208] In step S1309, the CPU 201A determines the emotion of the first user based on the body information of the first user controlled in step S1308 (performs emotion judgment again). The CPU 201A sends information related to the emotion of the first user to the server PC 101.

[0209] In step S1310, the CPU 306 receives the information related to the emotion of the first user.

[0210] The processing of subsequent steps S1311 to S1314 is a loop process, and this loop process is repeated the same number of times as the number of second users who participate in the desired space and view the video of the desired space.

[0211] In step S1311, the CPU 306 controls the usage avatar of the first user based on the information related to the emotion of the first user received in step S1310. For example, the CPU 306 controls the facial expression or movement (gesture) of the usage avatar to reflect the emotion of the first user.

[0212] Since steps S1312 to S1315 are similar to steps S613 to S616 according to the first embodiment, detailed description will not be repeated.

[0213] As described above, according to the fourth embodiment, the facial expression of the avatar etc. can be adjusted to the facial expression etc. that conforms to the communication purpose.

[0214] Note that in step S1308, when the body information obtained in step S1307 satisfies a specific condition, the CPU 201A can issue a warning (notification) to the first user. For example, when the CPU 201A determines that the negative emotion value (emotion value such as anxiety etc.) indicated by the emotion based on the body information exceeds the threshold, the CPU 201A can issue a warning (notification) to the first user. Alternatively, when the CPU 201A determines that the value indicated by the body information obtained in step S1307 (for example, blood pressure, heart rate or body temperature) exceeds the threshold, the CPU 201A can issue a warning (notification) to the first user. In this case, for example, the CPU 201A issues a warning to the first user before sending the body information to the server PC 101 in step S1309 to ask "Can the body information be sent?" or "Can the body information be changed?". The CPU 201A can display a display item for indicating the warning on the display 202, or can output a sound for indicating the warning from the speaker 211.

[0215] Although the warning is determined based on the body information before changing (controlling) the body information in step S1308, alternatively, the warning can be issued based on the body information after the change in step S1308.

[0216] Note that although the user terminal 102A changes the user's body information in step S1308 so that emotion judgment can be made based on the communication purpose etc., alternatively, the emotion information can be directly changed based on the communication purpose and related information.

[0217] In the fourth embodiment, the user terminal 102A sends the emotion information judged based on the user's body information to the server PC 101. However, the server PC 101 can perform the emotion judgment of the user and control the facial expression of the avatar based on the user's body information sent by the user terminal 102A.

[0218] As described above, according to the fourth embodiment, emotion information is controlled for the purpose of communication in a virtual space. Thus, for example, as shown in (B) of Figure 12 on the avatar 1202 of the user C in the virtual space viewed by the negotiation partner B, anxiety and tension are not shown and a calm facial expression is presented. In other words, the communication system can display an avatar with a more appropriate facial expression in a business negotiation.

[0219] In addition, the description "advance to step S1 when A is above B, but advance to step S2 when A is less than (below) B" presented above can be replaced with "advance to step S1 when A is greater than (above) B, but advance to step S2 when A is below B". Conversely, "advance to step S1 when A is greater than (above) B, but advance to step S2 when A is below B" can be replaced with "advance to step S1 when A is above B, but advance to step S2 when A is less than (below) B". Thus, unless there is an inconsistency, the expression "above A" can be replaced with "A or greater than (above; longer than; more than) A", or can be replaced with "greater than (above; longer than; more than) A". On the other hand, the expression "below A" can be replaced with "A or less than (below; shorter than; less than) A", or can be replaced with "less than (below; shorter than; less than) A". In addition, "greater than (above; longer than; more than) A" can be replaced with "above A", and "less than (below; shorter than; less than) A" can be replaced with "below A".

[0220] Although the present invention has been described in detail based on the preferred embodiments of the present invention, the present invention is not limited to the specific embodiments, and various modes without departing from the scope of the present invention are also included in the present invention. A part of the above embodiments can be appropriately combined with each other.

[0221] Note that each functional unit in the above embodiments (each modification) may be separate hardware, or may not be separate hardware. The functions of two or more functional units can be implemented by common hardware. Each function among the multiple functions of a single functional unit can be implemented by separate hardware. Two or more functions of a single functional unit can be implemented by common hardware. In addition, each functional unit can be implemented by hardware such as an ASIC, an FPGA, or a DSP, or may not be implemented by hardware such as an ASIC, an FPGA, or a DSP. For example, a device may include a processor and a memory (storage medium) for storing a control program. Further, at least a part of the functions of the functional units included in the device can be implemented by causing the processor to read the control program from the memory and execute the control program.

[0222] (Other Embodiments)

[0223] The present invention can also be implemented by supplying a program for implementing one or more functions of the above-described embodiments to a system or device via a network or a storage medium and causing one or more processors in a computer of the system or device to read and execute the program. In addition, the present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0224] The present invention is not limited to the above-described embodiments, and various modifications and changes can be made thereto without departing from the spirit and scope of the present invention. Therefore, the appended claims are attached to disclose the scope of the present invention.

[0225] This application claims priority based on Japanese Patent Application No. 2022-190101 filed on November 29, 2022, and the entire content disclosed therein is incorporated by reference.

[0226] [List of Reference Numerals]

[0227] 101 Server PC

[0228] 102 User Terminal

[0229] 201 CPU

[0230] 306 CPU

Claims

1. A system for enabling communication between a first user and a second user, the system comprising: An acquisition unit configured to acquire real-time information of the first user; And A control unit configured to control, based on the purpose of the communication, the reflection of the real-time information of the first user in the first avatar on a display device belonging to the second user and displaying a virtual space including the first avatar.

2. The system according to claim 1, wherein The real-time information at least includes information related to the body.

3. The system according to claim 2, wherein The real-time information includes information of at least one of voice, facial expression, blood pressure, heart rate, stress level, body temperature, sweating, brain waves, pulse rate, posture, and movement.

4. The system according to any one of claims 1 to 3, wherein The control unit reflects the real-time information of the first user in the first avatar by emphasizing or suppressing the real-time information based on the purpose of the communication.

5. The system according to any one of claims 1 to 4, wherein The control unit generates an image of a virtual space including the first avatar and controls the display device to display the image of the virtual space.

6. The system according to any one of claims 1 to 5, wherein The control unit controls the reflection of the real-time information of the first user in the first avatar based on the purpose of the communication and a reflection setting configured by the first user or the second user, and The reflection setting is a setting that correlates the purpose of the communication with the degree of reflection of the real-time information of the first user in the first avatar.

7. The system according to any one of claims 1 to 6, wherein The control unit controls the reflection of the real-time information of the first user in the first avatar based on the disease information of the first user and the purpose of the communication.

8. The system according to claim 7, wherein The control unit: Based on the disease information of the first user and the real-time information of the first user, determines whether the symptoms indicated by the disease information have occurred in the first user; And Based on the judgment result regarding whether the symptoms have occurred, controls the reflection of the real-time information of the first user in the first avatar.

9. The system according to any one of claims 1 to 8, wherein The control unit determines the purpose of the communication based on the information used to form the virtual space.

10. The system according to any one of claims 1 to 9, wherein The control unit determines the purpose of the communication based on information or an avatar related to at least one user account among multiple users in a group participating in the virtual space.

11. The system according to any one of claims 1 to 10, wherein The control unit determines the purpose of the communication based on information related to the tool used by the first user.

12. The system according to any one of claims 1 to 8, wherein The purpose of the communication is the purpose input by at least one user among multiple users who participate in the group in the virtual space.

13. The system according to any one of claims 1 to 12, wherein the control unit controls the reflection of the real-time information of the first user in the first avatar based on the purpose of the communication and the nationalities of multiple users who participate in the group in the virtual space.

14. The system according to any one of claims 1 to 13, wherein the control unit controls the reflection of the real-time information of the first user in the first avatar based on the purpose of the communication and the relationships among multiple users who participate in the group in the virtual space.

15. The system according to any one of claims 1 to 14, wherein the control unit controls the reflection of the real-time information of the first user in the first avatar based on the purpose of the communication and the role of the second user.

16. The system according to claim 15, wherein the control unit obtains information related to the role of the second user based on information related to the account of the second user or the operation of the first user.

17. The system according to claim 15, wherein the control unit obtains information related to the role of the second user from an external system.

18. The system according to any one of claims 1 to 17, further comprising: a warning unit configured to issue a warning to the first user before the control unit reflects the real-time information of the first user in the first avatar when the real-time information of the first user meets a specific condition.

19. A system control method for implementing communication between a first user and a second user, the system control method comprising: an obtaining step for obtaining the real-time information of the first user; and a control step for controlling the reflection of the real-time information of the first user in the first avatar on a display device of a virtual space that belongs to the second user and displays the first avatar including the first user, based on the purpose of the communication.

20. A program for causing a computer to function as each unit of the system according to any one of claims 1 to 18.

Images