Terminal device, program, and method of operating the terminal device

Abstract

To contribute to the convenience of a user who makes a call in a virtual space.SOLUTION: A terminal device includes a communication unit, an output unit, and a control unit that communicates with terminal devices of a plurality of communication partners through the communication unit, and outputs an image of a virtual space including images of the plurality of communication partners and voices of the plurality of communication partners through the output unit, and the control unit outputs images or voices of the communication partners of a plurality of groups to which the communication partners belong in a manner different from that of the other communication partners, and transmits information for outputting the communication partner's image or sound to the terminal devices of the communication partners of the plurality of groups in a manner different from that of the other communication partners.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a terminal device, a program, and an operation method of a terminal device.

Background Art

[0002] A method is known in which computers at multiple locations communicate via a network and multiple users talk to each other in a virtual space on the network. Regarding such calls on the network, various techniques have been proposed to improve user convenience. For example, Patent Document 1 discloses a technique for adjusting a parameter of a call volume according to the distance from another user in a virtual space in a call system in which multiple users can talk while viewing each other's images.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is room for further improvement in user convenience when making a call in a virtual space.

[0005] The present disclosure provides a terminal device and the like that contribute to the convenience of users making calls in a virtual space.

Means for Solving the Problems

[0006] The terminal device in this disclosure includes a communication unit, an output unit, and a control unit that communicates with the terminal devices of multiple call partners via the communication unit and outputs images of a virtual space including images of the multiple call partners and the voices of the multiple call partners via the output unit. The control unit outputs images or voices of call partners in multiple groups to which the caller belongs in a manner different from that of other call partners, and sends information to the terminal devices of the call partners in the multiple groups to cause them to output images or voices of the caller in a manner different from that of the terminal devices of other call partners.

[0007] The program for the terminal device in this disclosure is executed by the control unit of a terminal device having a communication unit, an output unit, and a control unit, and causes the control unit to perform the steps of communicating with the terminal devices of a plurality of callers via the communication unit, and outputting an image of a virtual space including images of the plurality of callers and the voices of the plurality of callers via the output unit, further causing the control unit to perform the steps of outputting images or voices of callers from a plurality of groups to which the caller belongs in a manner different from that of other callers, and sending information to the terminal devices of the callers in the plurality of groups to output images or voices of the caller in a manner different from that of the terminal devices of other callers.

[0008] The method of operating a terminal device in this disclosure is a method of operating a terminal device having a communication unit, an output unit, and a control unit, the method comprising the steps of: the control unit communicating with the terminal devices of a plurality of callers by the communication unit, and outputting an image of a virtual space including images of the plurality of callers and the voices of the plurality of callers by the output unit; and the control unit further outputting images or voices of callers from a plurality of groups to which the caller belongs in a manner different from that of other callers, and sending information to the terminal devices of the callers from the plurality of groups to cause them to output images or voices of the caller in a manner different from that of the terminal devices of other callers. [Effects of the Invention]

[0009] The terminal devices and other equipment described in this disclosure will make it possible to improve the convenience of users who make calls in a virtual space. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing an example configuration of a virtual event delivery system. [Figure 2] This is a sequence diagram illustrating an example of the operation of a virtual event provisioning system. [Figure 3A] This is a flowchart illustrating an example of terminal device operation. [Figure 3B] This is a flowchart illustrating an example of terminal device operation. [Figure 4] This is a diagram showing an example of a virtual space image. [Figure 5A] This is a flowchart illustrating an example of terminal device operation. [Figure 5B] This is a flowchart illustrating an example of terminal device operation. [Figure 6A] This is a diagram showing an example of a virtual space image. [Figure 6B] This is a diagram showing an example of a virtual space image. [Figure 7A] This is a diagram showing an example of a virtual space image. [Figure 7B] This is a diagram showing an example of a virtual space image. [Modes for carrying out the invention]

[0011] The embodiments will be described below.

[0012] FIG. 1 is a diagram showing a configuration example of a virtual event providing system 1 in an embodiment. The virtual event providing system 1 includes a server device 10 and a plurality of terminal devices 12 that are communicably connected to each other via a network 11. The virtual event providing system 1 is a system for providing an event, that is, a virtual event, in a virtual space in which a user can participate using the terminal device 12. The virtual event is an event in which a plurality of users can communicate with each other by speaking or the like in the virtual space, and each user is represented by an image such as a 3D model representing the user. The event in the present embodiment is a discussion about an arbitrary theme, a party for free conversation between users, or the like.

[0013] The server device 10 is, for example, a server computer that belongs to a cloud computing system or other computing system and functions as a server that implements various functions. The server device 10 may be configured by two or more server computers that are communicably connected and operate in cooperation. The server device 10 executes transmission and reception of information necessary for providing a virtual event and information processing.

[0014] The terminal device 12 is an information processing device having a communication function, and is used by a user who participates in a virtual event provided by the server device 10. The terminal device 12 is, for example, an information processing terminal such as a smartphone or a tablet terminal, or an information processing device such as a personal computer.

[0015] The network 11 is, for example, the Internet, but includes an ad hoc network, a LAN (Local Area Network), a MAN (Metropolitan Area Network), or other network or any combination thereof.

[0016] In the following description, when explaining the call between the users of the terminal device 12 from the perspective of one terminal device 12, the user of that terminal device 12 is referred to as the caller, and the user of another terminal device 12 is referred to as the callee.

[0017] In this embodiment, the terminal device 12 includes a communication unit 111, an output unit 116, and a control unit 113. The control unit 113 communicates with the terminal devices 12 of a plurality of call partners via the communication unit 111, and outputs an image of a virtual space including images of the plurality of call partners and voices of the plurality of call partners via the output unit 116. Further, the control unit 113 outputs the images or voices of the call partners in a plurality of groups to which the caller belongs in a manner different from that of other call partners, and transmits information for causing the terminal devices 12 of the call partners in the plurality of groups to output the image or voice of the caller in a manner different from that of the terminal devices 12 of other call partners. According to the terminal device 12, a caller can select a desired call partner from a plurality of call partners to form a group, or participate in a group of desired call partners, and make a call so that the call cannot be heard by the call partners outside the group. Hereinafter, such a call is referred to as a group call. In this embodiment, since a caller can participate in a plurality of groups for a group call, it is possible to improve the convenience of a user making a call in a virtual space.

[0018] The configurations of the server device 10 and the terminal device 12 will be described in detail below.

[0019] The server device 10 includes a communication unit 101, a storage unit 102, a control unit 103, an input unit 105, and an output unit 106. These configurations are appropriately arranged in two or more computers when the server device 10 is composed of two or more server computers.

[0020] The communication unit 101 includes one or more communication interfaces. The communication interface is, for example, a LAN interface. The communication unit 101 receives information used for the operation of the server device 10, and transmits information obtained by the operation of the server device 10. The server device 10 is connected to the network 11 via the communication unit 101, and performs information communication with the terminal device 12 via the network 11.

[0021] The storage unit 102 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these, which function as main memory, auxiliary memory, or cache memory. The semiconductor memory is, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The RAM is, for example, SRAM (Static RAM) or DRAM (Dynamic RAM). The ROM is, for example, EEPROM (Electrically Erasable Programmable ROM). The storage unit 102 stores information used for the operation of the server device 10 and information obtained by the operation of the server device 10.

[0022] The control unit 103 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processors are, for example, general-purpose processors such as CPUs (Central Processing Units) or dedicated processors such as GPUs (Graphics Processing Units) specialized for specific processing. The dedicated circuits are, for example, FPGAs (Field-Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits). The control unit 103 controls each part of the server device 10 and performs information processing related to the operation of the server device 10.

[0023] The input unit 105 includes one or more input interfaces. The input interfaces are, for example, physical keys, capacitive keys, pointing devices, touchscreens integrated with a display, or microphones that accept voice input. The input unit 105 accepts operations to input information used for the operation of the server device 10 and sends the input information to the control unit 103.

[0024] The output unit 106 includes one or more output interfaces. The output interfaces are, for example, a display or a speaker. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The output unit 106 outputs information obtained from the operation of the server device 10.

[0025] The functions of the server device 10 are realized by the execution of a control program by a processor included in the control unit 103. The control program is a program that causes the computer to function as the server device 10. In addition, some or all of the functions of the server device 10 may be realized by a dedicated circuit included in the control unit 103. Furthermore, the control program may be stored in a non-transient recording / storage medium readable by the server device 10, and the server device 10 may read it from the medium.

[0026] The terminal device 12 includes a communication unit 111, a storage unit 112, a control unit 113, an input unit 115, an output unit 116, and an imaging unit 117.

[0027] The communication unit 111 includes a communication module compatible with wired or wireless LAN standards, a module compatible with mobile communication standards such as LTE, 4G, and 5G, etc. The terminal device 12 is connected to the network 11 via the communication unit 111 through a nearby router device or mobile communication base station, and communicates information with the server device 10, etc. via the network 11.

[0028] The storage unit 112 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. The semiconductor memory is, for example, RAM or ROM. The RAM is, for example, SRAM or DRAM. The ROM is, for example, EEPROM. The storage unit 112 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 112 stores information used in the operation of the control unit 113 and information obtained by the operation of the control unit 113.

[0029] The control unit 113 has, for example, one or more general-purpose processors such as a CPU or MPU (Micro Processing Unit), or one or more dedicated processors such as a GPU specialized for specific processing. Alternatively, the control unit 113 may have one or more dedicated circuits such as FPGAs or ASICs. The control unit 113 comprehensively controls the operation of the terminal device 12 by operating according to a control and processing program, or by operating according to an operating procedure implemented as a circuit. The control unit 113 then sends and receives various information with the server device 10, etc. via the communication unit 111 and executes the operations according to this embodiment.

[0030] The input unit 115 includes one or more input interfaces. These input interfaces include, for example, physical keys, capacitive keys, pointing devices, touchscreens integrated with displays, and proximity sensors such as infrared sensors for detecting user gestures. The input interfaces may also include a microphone for receiving voice input and a camera for capturing images. Furthermore, the input interfaces may include a scanner or camera for scanning image codes and an IC card reader. The input unit 115 accepts operations for inputting information used in the operation of the control unit 113 and sends the input information to the control unit 113.

[0031] The output unit 116 includes one or more output interfaces. The output interfaces include, for example, a display and a speaker. The display is, for example, an LCD or an organic EL display. The output unit 116 outputs information obtained by the operation of the control unit 113.

[0032] The imaging unit 117 includes a camera that captures an image of a subject using visible light, and a distance measuring sensor that measures the distance to the subject and acquires a distance image. The camera captures images of the subject at, for example, 15 to 30 frames per second to generate a moving image consisting of a series of captured images. The distance measuring sensor includes a ToF (Time Of Flight) camera, LiDAR (Light Detection And Ranging), and a stereo camera, and generates a distance image of the subject that includes distance information. The imaging unit 117 sends the captured image and the distance image to the control unit 113.

[0033] The functions of the control unit 113 are realized by the execution of a control program by the processor included in the control unit 113. The control program is a program that causes the processor to function as the control unit 113. In addition, some or all of the functions of the control unit 113 may be realized by a dedicated circuit included in the control unit 113. Furthermore, the control program may be stored in a non-transient recording / storage medium readable by the terminal device 12, and the terminal device 12 may read it from the medium.

[0034] In this embodiment, the control unit 113 acquires an image of the caller and a distance image using the imaging unit 117, and collects the caller's spoken voice using the microphone of the input unit 115. The control unit 113 encodes the image of the caller and the distance image for generating a 3D model representing the caller, and audio information for reproducing the caller's voice, to generate encoded information. The control unit 113 may perform arbitrary processing on the image, etc. (for example, resolution change and cropping) during encoding. The control unit 113 sends the encoded information to other terminal devices 12 via the server device 10 using the communication unit 111. The control unit 113 also receives encoded information sent from other terminal devices 12 via the server device 10 using the communication unit 111. When the control unit 113 decodes the encoded information received from other terminal devices 12, it uses the decoded information to generate a 3D model representing the caller using the other terminal device 12, and places the 3D model in the virtual space. The control unit 113 may further generate a 3D model representing the caller using the caller's captured image and depth image, and place it in the virtual space. When generating the 3D model, the control unit 113 generates a polygon model using the depth image and then applies texture mapping using the captured image to the polygon model to generate the 3D model. However, the method for generating the 3D model is not limited to the example shown here and any method can be used. When the control unit 113 generates a virtual space image for output, including the 3D model from a predetermined viewpoint in the virtual space, by rendering, the output unit 116 displays the virtual space image and outputs the caller's spoken voice based on the caller's voice information. Through such operation of the control unit 113, the caller of the terminal device 12 can participate in the virtual event in real time and communicate with the caller.

[0035] Figure 2 is a sequence diagram illustrating the operation procedure of the virtual event provision system 1. This sequence diagram shows the procedure for the coordinated operation of the server device 10 and multiple terminal devices 12 (referred to as terminal devices 12A and 12B for convenience). This procedure is, for example, the procedure executed when a caller using terminal device 12A invites a caller using terminal device 12B to a virtual event as the administrator and participant of the virtual event. When inviting multiple callers, the procedure for terminal device 12B shown here is executed by each of the callers' terminal devices 12B, or by each of the multiple terminal devices 12B and the server device 10.

[0036] The steps related to various information processing in the server device 10 and terminal device 12 in Figure 2 are performed by their respective control units 103 and 113. Furthermore, the steps related to sending and receiving various information in the server device 10 and terminal device 12 are performed by the respective control units 103 and 113 sending and receiving information to each other via the communication units 101 and 111, respectively. In the server device 10 and terminal device 12, the control units 103 and 113 respectively store the information to be sent and received in the storage units 102 and 112 as appropriate. In addition, the control unit 113 of the terminal device 12 receives input of various information via the input unit 115 and outputs various information via the output unit 116.

[0037] In step S200, terminal device 12A receives input of virtual event configuration information from the caller. The configuration information includes the virtual event schedule, discussion topic, and participant list. The participant list includes participant names and each participant's email address. Then, in step S201, terminal device 12A sends the configuration information to server device 10. Server device 10 receives the information sent from terminal device 12A. For example, terminal device 12A accesses the site for conducting the virtual event provided by server device 10, obtains the configuration information input screen, and displays the input screen to user A. Then, when the caller enters the configuration information on the input screen, the configuration information is sent to server device 10.

[0038] In step S202, the server device 10 sets up a virtual event based on the configuration information. The control unit 103 associates the virtual event information with the information of the planned participants and stores it in the storage unit 102.

[0039] In step S203, the server device 10 sends authentication information to the terminal device 12B. The authentication information is information used to identify and authenticate user B using the terminal device 12B, and includes information such as an ID and passcode used when participating in a virtual event. This information is sent, for example, as an attachment to an email. The terminal device 12B receives the information sent from the server device 10.

[0040] In step S205, terminal device 12B sends the authentication information and participation application information received from server device 10 to server device 10. The caller operates terminal device 12B to apply to participate in the virtual event using the authentication information sent from server device 10. For example, terminal device 12B accesses the site for the virtual event provided by server device 10, obtains the input screen for authentication information and participation application information, and displays the input screen to the caller. Then, terminal device 12B receives the information entered by the caller and sends it to server device 10.

[0041] In step S206, the server device 10 authenticates the caller and completes the participant registration. The memory unit 102 stores the identification information of the terminal device 12B and the identification information of the caller in association.

[0042] In steps S208 and S209, the server device 10 sends an event start notification to terminal devices 12A and 12B, respectively. Upon receiving the information from the server device 10, terminal devices 12A and 12B each begin imaging the caller and the other party, and collecting their spoken voice, respectively.

[0043] In step S210, a virtual event is performed by terminal devices 12A and 12B via the server device 10. Terminal devices 12A and 12B each send and receive information for generating 3D models representing the caller and the other party, as well as spoken audio information, via the server device 10. Terminal devices 12A and 12B also output images of the virtual event, including 3D models representing themselves and the other party, and the other party's spoken audio to the caller and the other party, respectively.

[0044] Figures 3A and 3B are flowcharts illustrating the operating procedures of terminal device 12 related to the execution of a virtual event. The procedures shown here are common to terminal devices 12A and 12B, and will be explained without distinguishing between terminal devices 12A and 12B.

[0045] Figure 3A illustrates the operation procedure of the control unit 113 when the terminal device 12 sends out information for generating a 3D model of the caller and information for the spoken voice.

[0046] In step S302, the control unit 113 uses the imaging unit 117 to capture a visible light image of the participant at an arbitrarily set frame rate and acquire a distance image, and the input unit 115 to collect the voice of the caller's speech. The control unit 113 acquires the visible light image and distance image from the imaging unit 117 and acquires voice information from the input unit 115.

[0047] In step S304, the control unit 113 encodes the captured image, distance image, and audio information, and generates encoded information.

[0048] In step S306, the control unit 113 packets the encoded information using the communication unit 111 and sends it to the server device 10 for other terminal devices 12.

[0049] When the control unit 113 receives information input in response to an operation by the caller to interrupt imaging / sound collection or to exit a virtual event (Yes in S308), it terminates the processing procedure shown in Figure 3A. If it does not receive information corresponding to an operation to interrupt or exit (No in S308), it executes steps S302 to S306, sending information for generating a 3D model representing the caller and information for outputting audio to other terminal devices 12.

[0050] Figure 3B illustrates the operation procedure of the control unit 113 when terminal device 12 outputs an image of a virtual event including the caller and the caller's voice. When the control unit 113 receives packets sent by other terminal devices 12 performing the procedure in Figure 3A via the server device 10, it executes steps S310 to S313. When receiving information from multiple caller terminal devices 12, the control unit 113 executes steps S310 to S313 for each of the other terminal devices 12. Furthermore, when the control unit 113 acquires the caller's captured image, distance image, and spoken voice, it executes steps S310 to S313.

[0051] In step S310, the control unit 113 decodes the encoded information contained in the packet received from the other terminal device 12 to obtain the captured image, distance image, and audio information. Also, when executing step S302, the control unit 113 obtains the captured image and distance image of the caller from the imaging unit 117 and the audio information from the input unit 115.

[0052] In step S312, the control unit 113 generates a 3D model representing each participant based on the captured image and the depth image.

[0053] In step S313, the control unit 113 places 3D models representing the caller and the other party in the virtual space where the virtual event is held. The storage unit 112 stores in advance the coordinate information of the virtual space and the coordinate information of where the 3D models of the caller and the other party should be placed, for example, in the order in which they are authenticated.

[0054] In step S314, the control unit 113 renders and generates a virtual space image by capturing multiple 3D models placed in the virtual space from a virtual viewpoint.

[0055] In step S316, the control unit 113 displays a virtual space image and outputs audio via the output unit 116. Specifically, the control unit 113 outputs information to the output unit 116 for displaying an image of an event in which a 3D model is placed in the virtual space, and the output unit 116 outputs the virtual space image and the voice of the caller. For example, as shown in Figure 4, the output unit 116 displays a virtual space image 400 that includes the caller 40 represented by a 3D model and the other callers 41-46.

[0056] The control unit 113 repeatedly executes steps S310 to S316, allowing the caller to view a video of a virtual space image including a 3D model of the other party while listening to the other party's speech. The control unit 113 can also move the position of the caller 40 in the virtual space image 400 in response to the caller's actions. These actions may include dragging on a touch panel or pointing device, or using gestures. The control unit 113 sends information regarding the movement of the 3D model, such as the direction and amount of movement, to other terminal devices 12. This allows the other terminal devices 12 to move the caller 40 in the virtual space image 400 displayed to the other party. The control unit 113 may also move the other party 41 to 46 in response to the movement information sent from the other terminal devices 12.

[0057] Figures 5A and 5B are flowcharts illustrating the operation procedure of terminal device 12 related to the execution of a group call in a virtual event. The procedure shown here is common to terminal devices 12A and 12B, and will be explained without distinguishing between terminal devices 12A and 12B.

[0058] Figure 5A illustrates the operation procedure when the control unit 113 in the terminal device 12 performs a process for a caller to form a group consisting of desired call partners. The procedure in Figure 5A is performed, for example, in response to an operation by the caller to instruct group formation.

[0059] In step S500, the control unit 113 receives a group formation instruction. The control unit 113 receives input from the input unit 115 in response to the caller's operation. The caller selects a desired caller in the virtual space image, for example, by tapping or clicking with a pointing device, thereby instructing the control unit 113 to form a group with the selected caller.

[0060] In step S501, the control unit 113 notifies that a group has been formed. The control unit 113 stores information identifying the call partners selected by the caller in association with each other, and notifies the terminal devices 12 of the call partners within the group and the terminal devices 12 of other call partners outside the group of the group's formation. The notification includes information identifying the callers and call partners within the group. The control unit 113 may also require that the terminal devices 12 of the call partners within the group consent to join the group as a condition for proceeding to the next step.

[0061] In step S502, the control unit 113 displays the formed group. The control unit 113 displays the callers and call partners within the group in a different manner from other call partners outside the group. For example, in the virtual space image 400 shown in Figure 6A, the callers 40, call partners 41 and 42 within the group are displayed in a different manner from other call partners 43-46 outside the group. Different display manners include adding effects such as illumination and shading, emphasizing contours, increasing the number of polygons, and enlarging the size. Alternatively, the control unit 113 may place the callers 40 and call partners 41 and 42 within the group close together and surround them with a frame 60. However, the display manner is not limited to those shown here, as long as the callers 40 and call partners 41 and 42 within the group and the other call partners 43-46 can be distinguished.

[0062] In step S503, the control unit 113 starts a group call. The control unit 113 outputs the voices of the callers within the group in a different manner than the voices of other callers outside the group. For example, in the example of the virtual space image 400 in Figure 6A, the control unit 113 outputs the voices of callers 41 and 42 within the group at a higher volume than the voices of other callers 43-46 outside the group. Alternatively, the control unit 113 may mute the voices of other callers 43-46 outside the group. In that case, the control unit 113 may display strings corresponding to the voices of other callers 43-46 outside the group. This makes it easier for callers to concentrate on the group call.

[0063] Figure 5B illustrates the operation procedure when the control unit 113 in terminal device 12 performs a process for a caller to join a group formed by the other party to the call. The procedure in Figure 5B is performed, for example, when step S501 in Figure 5A is performed in another terminal device and a group formation notification is sent, and the control unit 113 responds to the notification.

[0064] In step S504, the control unit 113 receives a notification of group formation. Based on the notification, the control unit 113 identifies the callers to be included in the group. At this point, the control unit 113 may output a message to the callers to confirm whether they agree to join the group, receive an input indicating their acceptance or rejection, and return that information to the other terminal devices 12.

[0065] In step S505, the control unit 113 joins the group. The control unit 113 identifies the callers within the group and stores the information of the identified callers in association with each other.

[0066] In step S506, the control unit 113 displays the group in which the user has joined. The control unit 113 displays the callers and other callers within the group in a different manner from other callers outside the group, for example, as shown in the example in Figure 6A.

[0067] In step S508, the control unit 113 initiates a group call. The control unit 113 outputs the voice of the callers within the group in a different manner than the voice of other callers outside the group.

[0068] The procedures shown in Figures 5A and 5B above are executed in each terminal device 12, thereby enabling a group call by multiple terminal devices 12. Furthermore, the terminal devices 12 of callers outside the group will display the callers forming the group in a different manner than other callers, based on notifications from the terminal devices 12 of the callers forming the group. This allows callers outside the group to recognize the formation of the group.

[0069] The procedure in Figure 5A is also performed when a caller selects a caller in a group they are not a part of. For example, the control unit 113 displays a virtual space image 400 as shown in Figure 6B. Here, a state is shown where a group has been formed by callers 41 and 42, in which caller 40 is not a part. The formation of a group is indicated by the fact that callers 41 and 42 are displayed in a different manner from the other callers 43-46, such as by being surrounded by a frame 61. The control unit 113 accepts input from the caller to select callers 41 and 42, for example, by tapping or using a gesture. Alternatively, the caller can select within the area of ​​the frame 61 by tapping or using a gesture. In this case, in step S501, the control unit 113 stores information identifying the callers selected by the caller in association with each other, and notifies the terminal devices 12 of the callers in the group and the terminal devices 12 of other callers outside the group of the group update. In this way, the caller can join an already existing group and perform a group call.

[0070] Furthermore, the procedure shown in Figure 5A is also performed when a caller selects a caller from outside the group while already being a member of the group. In this case, in step S501, the control unit 113 stores the information of the callers already in the group and the information identifying the newly selected caller in correspondence with each other, and notifies the terminal devices 12 of the callers within the group and the terminal devices 12 of the other callers outside the group of the group update. In this way, the caller can add a new caller to an existing group and perform a group call.

[0071] The procedure in Figure 5B is also performed when the caller selects a caller from among those who are already in the group. The control unit 113 stores information that identifies the callers to be included in the group from the group formation notification. In this way, the caller can join a group they are already in and perform a group call.

[0072] Furthermore, the procedure shown in Figure 5B is also executed when a caller is already a member of a group and is selected from a caller outside the group. For example, in step S505, the control unit 113 updates the group and joins the new caller. The control unit 113 identifies the callers within the group and the new caller, and stores the information of the identified callers in correspondence with each other. In this way, the caller can update the group they are already a member of, join the updated group, and perform a group call.

[0073] In this embodiment, the terminal device 12 further enables the caller to make group calls in multiple other groups. When the procedures in Figures 5A and 5B are performed by the terminal device 12 and the caller is making a group call, if the caller performs an operation on the terminal device 12 to form another group with another caller or to join another group, the terminal device 12 will perform the procedures in Figures 5A and 5B with the other caller or group.

[0074] Figure 7A shows an example of a virtual space image 400 when a caller forms another group or joins another group while a group call is in progress. Here, the state is shown when caller 40 is in a group call with call partners 41 and 42 in a group enclosed by a frame 60, and then forms another group with other call partners 43 and 44, or joins the group of call partners 43 and 44. In this case, instead of updating the group containing call partners 41 and 42 to a group containing call partners 43 and 44, the caller performs an operation to choose to join different groups simultaneously. For example, the caller performs an operation such as tapping or a gesture to select such a function from a pop-up menu. When the caller selects call partners 43 and 44, or selects their group, the control unit 113 displays the caller's avatar 40a, and the group formed and displayed by the avatar 40a and call partners 43 and 44 is displayed. Here, the group consisting of the avatar 40a and the callers 43 and 44 is shown enclosed by a border 70, and the callers 45 and 46, who are not included in any group, are displayed in a different manner. In addition, the avatar 40a of the caller participating in the group is displayed on the respective terminal devices 12 of callers 43 and 44, so that each caller can recognize the participation of the caller.

[0075] In this embodiment, when a caller participates in multiple groups simultaneously, the terminal device 12 outputs the audio of multiple group calls to the caller. However, the terminal device 12 selectively sends the caller's voice information to the terminal device 12 of the primary group's call partner, but does not send the caller's voice information to the terminal devices 12 of the other secondary groups' call partners. For example, in the example in Figure 7A, when a caller makes a group call with call partners 41 and 42 as the primary group and call partners 43 and 44 as secondary groups, the control unit 113 sends the caller's voice information to the respective terminal devices 12 of call partners 41 and 42, and receives the voice information of call partners 41 and 42 from their respective terminal devices 12. On the other hand, the control unit 113 does not send the caller's voice information to the respective terminal devices 12 of call partners 43 and 44, but receives the voice information of call partners 43 and 44 from their respective terminal devices 12. This allows the caller to understand the content of conversations in multiple groups while avoiding confusion. The control unit 113 may, in group calls within subordinate groups, display a string of characters corresponding to the voice instead of outputting the voice of the other party.

[0076] Furthermore, as shown in Figure 7A, the control unit 113 displays the doppelganger 40a in a manner different from that of the caller 40. This makes it easier for the caller to determine the hierarchy of groups, for example, by designating the group that caller 40 belongs to as the primary group and the group that the doppelganger 40a belongs to as a secondary group. In addition, the control unit 113 may display the callers 41 and 42 of the primary group in a manner different from that of the callers 43 and 44 of the secondary group, for example by using effects such as illumination, shading, or contour enhancement.

[0077] Furthermore, the control unit 113 can appropriately change the primary group in response to operations performed by the caller. Such operations include, for example, tapping or gesture operations to select a change in the primary / slave status of the group from a pop-up menu.

[0078] Figure 7B is an example of the virtual space image 400 displayed on the terminal device 12 of the call recipient 44 in Figure 7A. Here, the call recipient 44's group displays the caller's avatar 40a and the call recipient 44. On the other hand, the groups of caller 40 and call recipients 41 and 42 for caller 40 are also displayed. In this way, the call recipient 44 can also be aware of the existence of the avatar 40a, thus reducing the sense of unease in group calls.

[0079] In the above description, the caller and the person being called were represented by 3D models based on captured images, but they may also be represented by images such as avatars or characters generated without being based on captured images. In that case, terminal device 12 sends image information for the avatar, character, etc., instead of captured images, distance images, etc., to other terminal devices 12. By doing so, each terminal device 12 can display images of the caller and the person being called.

[0080] According to the above-described embodiment, callers can participate in group calls by joining multiple groups, thereby improving the convenience for users making calls in a virtual space.

[0081] As described above, embodiments have been explained based on various drawings and examples, but it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of this disclosure. For example, the functions, etc., included in each means, each step, etc., can be rearranged in a logically consistent manner, and multiple means, steps, etc., can be combined into one or divided. [Explanation of symbols]

[0082] 1. Virtual Event Delivery System 10 Server devices 11 Network 12 Terminal devices 101, 111 Communications Department 102, 112 Storage section 103, 113 Control Unit 105, 115 Input Sections 106, 116 Output section 117 Imaging Unit

Claims

1. A system having a server device and a plurality of terminal devices that can communicate with each other, Each terminal device communicates with the terminal devices of multiple call partners via the server device, and outputs an image of a virtual space including the image of the caller of each terminal device and the multiple call partners, as well as the audio of the multiple call partners. When a first terminal device used by a caller belonging to a group communicates with a second terminal device used by a caller belonging to the same group, the first and second terminal devices each output the caller's image in a manner that is more emphasized than the image of a caller not belonging to the group, or each output the caller's voice in a manner that is more emphasized than the voice of a caller not belonging to the group. system.

2. In claim 1, The first and second terminal devices output strings corresponding to the voices of callers belonging to the group. system.

3. A method for operating a system having a server device and a plurality of terminal devices that can communicate with each other, Each terminal device communicates with the terminal devices of multiple call partners via the server device, and outputs an image of a virtual space including the image of the caller of each terminal device and the multiple call partners, as well as the audio of the multiple call partners. When a first terminal device used by a caller belonging to a group communicates with a second terminal device used by a caller belonging to the same group, the first and second terminal devices each output the caller's image in a manner that is more emphasized than the image of a caller not belonging to the group, or each output the caller's voice in a manner that is more emphasized than the voice of a caller not belonging to the group. A method of operation that includes this.

4. In claim 3, A method of operation in which the first and second terminal devices output strings corresponding to the voices of callers belonging to the group.

Images