Terminal device, medium, and method of operation of a terminal device

Abstract

Disclosed is a terminal device, a medium, and a method for operating a terminal device. This contributes to the convenience of a user who makes a call in a virtual space. The terminal device has a communication section, an output section, and a control section. The communication section communicates with a plurality of terminal devices of call partners. The output section outputs an image of a virtual space including images of the plurality of call partners and voices of the plurality of call partners. The control section outputs images or voices of call partners of a plurality of groups to which a caller belongs in a different style from other call partners and sends information for outputting images or voices of the caller in a different style from terminal devices of other call partners to the terminal devices of the call partners of the plurality of groups.

Description

Technical Field

[0001] This disclosure relates to terminal devices, media, and methods of operating terminal devices. Background Technology

[0002] It is known that computers in multiple locations communicate via a network, and multiple users can talk to each other in a virtual space on the network. Various techniques have been proposed to improve user convenience when talking on such a network. For example, Patent Document 1 discloses a technique for adjusting the call volume parameter based on the distance to other users in the virtual space in a call system where multiple users can talk while viewing each other's images.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent No. 6849133 Summary of the Invention

[0006] There is room to further improve user convenience when making calls in virtual spaces.

[0007] This disclosure provides a convenient terminal device or the like that helps users make calls in virtual space.

[0008] The terminal device disclosed herein includes: a communication unit; an output unit; and a control unit. The communication unit communicates with the terminal devices of multiple call partners. The output unit outputs an image of a virtual space including the images of the multiple call partners and the voices of the multiple call partners. The control unit outputs the images or voices of call partners belonging to multiple groups of call partners in a style different from other call partners, and sends information to the terminal devices of the multiple groups of call partners for outputting the image or voice of the caller in a style different from other call partners' terminal devices.

[0009] The program of the media-stored terminal device in this disclosure is executed by the control unit of the terminal device having a communication unit, an output unit, and a control unit. The program causes the control unit to perform the following steps: communicating with the terminal devices of multiple call partners through the communication unit; and causing the output unit to output an image of a virtual space including the images of the multiple call partners and the voices of the multiple call partners. The program also causes the control unit to perform the following steps: outputting the images or voices of call partners in multiple groups to which the caller belongs in a different style from other call partners; and sending information to the terminal devices of the multiple groups of call partners for outputting the image or voice of the caller in a different style from other call partners' terminal devices.

[0010] The terminal device operation method disclosed herein includes a communication unit, an output unit, and a control unit, comprising: the control unit communicating with the terminal devices of multiple call partners through the communication unit, and outputting an image of a virtual space including images of the multiple call partners and the voices of the multiple call partners through the output unit; and the control unit further outputting the images or voices of call partners in multiple groups to which the caller belongs in a different style from other call partners, and sending information to the terminal devices of the multiple groups of call partners for outputting the image or voice of the caller in a different style from other call partners' terminal devices.

[0011] The terminal devices and the like disclosed herein can facilitate users making calls in virtual space. Attached Figure Description

[0012] Figure 1 This is a diagram illustrating an example of the structure of a virtual event provisioning system.

[0013] Figure 2 This is a timing diagram illustrating an example of the actions of the virtual event providing system.

[0014] Figure 3A This is a flowchart illustrating an example of the operation of a terminal device.

[0015] Figure 3B This is a flowchart illustrating an example of the operation of a terminal device.

[0016] Figure 4 This is a diagram showing an example of a virtual space image.

[0017] Figure 5A This is a flowchart illustrating an example of the operation of a terminal device.

[0018] Figure 5B This is a flowchart illustrating an example of the operation of a terminal device.

[0019] Figure 6A This is a diagram showing an example of a virtual space image.

[0020] Figure 6B This is a diagram showing an example of a virtual space image.

[0021] Figure 7A This is a diagram showing an example of a virtual space image.

[0022] Figure 7B This is a diagram showing an example of a virtual space image.

[0023] (Symbol Explanation)

[0024] 1: Virtual event providing system; 10: Server device; 11: Network; 12: Terminal device; 101, 111: Communication unit; 102, 112: Storage unit; 103, 113: Control unit; 105, 115: Input unit; 106, 116: Output unit; 117: Camera unit. Detailed Implementation

[0025] The implementation method is described below.

[0026] Figure 1 This diagram illustrates a structural example of a virtual event providing system 1 in one embodiment. The virtual event providing system 1 includes a server device 10 and multiple terminal devices 12 connected via a network 11 to communicate with each other. The virtual event providing system 1 is a system for providing events, or virtual events, in a virtual space that users can participate in using their terminal devices 12. Virtual events are events in a virtual space where multiple users can communicate with each other through voice or other means, and each user is represented by an image such as a 3D model. Events in this embodiment include seminars on any topic, gatherings for free conversation among users, etc.

[0027] Server device 10 may be a cloud computing system or other computing system, functioning as a server computer with various installed functions. Server device 10 may also be composed of two or more server computers connected and cooperating in a manner that allows for information communication. Server device 10 performs the sending and receiving of information required for providing virtual events and information processing.

[0028] Terminal device 12 is an information processing device with communication capabilities, used by users participating in virtual events provided by server device 10. Terminal device 12 is, for example, an information processing terminal such as a smartphone or tablet, or an information processing device such as a personal computer.

[0029] Network 11 is, for example, the Internet, but includes ad hoc networks, LANs (Local Area Networks), MANs (Metropolitan Area Networks), or other networks or any combination thereof.

[0030] In the following description, when discussing a call between users of terminal device 12 from the perspective of one terminal device 12, the user of that terminal device 12 is referred to as the caller, and the user of other terminal devices 12 is referred to as the caller.

[0031] In this embodiment, the terminal device 12 includes a communication unit 111, an output unit 116, and a control unit 113. The control unit 113 communicates with the terminal devices 12 of multiple call partners via the communication unit 111, and outputs images of the virtual space including images of the multiple call partners and their audio via the output unit 116. Furthermore, the control unit 113 outputs the images or audio of call partners in multiple groups to which the caller belongs in a different style than other call partners, sending information to the terminal devices 12 of the multiple groups regarding the output of the caller's image or audio in a different style than other call partners. According to the terminal device 12, the caller can select desired call partners from multiple call partners to form a group, or join a group of desired call partners and communicate with them in a way that prevents other call partners outside the group from hearing the conversation. Hereinafter, this type of communication is referred to as a group call. In this embodiment, the caller can join multiple groups for group calls, thus improving the convenience for users communicating in virtual space.

[0032] The structure of the server device 10 and the terminal device 12 is described in detail.

[0033] The server device 10 includes a communication unit 101, a storage unit 102, a control unit 103, an input unit 105, and an output unit 106. When the server device 10 is composed of two or more server computers, these structures are appropriately configured among the two or more computers.

[0034] The communication unit 101 includes one or more communication interfaces. These communication interfaces may be, for example, LAN interfaces. The communication unit 101 receives information used in the operation of the server device 10 and transmits information obtained through the operation of the server device 10. The server device 10 is connected to the network 11 via the communication unit 101 and communicates with the terminal device 12 via the network 11.

[0035] Storage unit 102 includes, for example, one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or combinations of at least two of these, functioning as main storage, auxiliary storage, or cache memory. Semiconductor memories are, for example, RAM (Random Access Memory) or ROM (Read Only Memory). RAM is, for example, SRAM (Static RAM) or DRAM (Dynamic RAM). ROM is, for example, EEPROM (Electrically Erasable Programmable ROM). Storage unit 102 stores information used in the operation of server device 10 and information obtained through the operation of server device 10.

[0036] The control unit 103 includes one or more processors, one or more dedicated circuits, or combinations thereof. The processor may be a general-purpose processor such as a CPU (Central Processing Unit) or a dedicated processor such as a GPU (Graphics Processing Unit) specialized for specific processing. The dedicated circuit may be, for example, a FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). The control unit 103 controls the various parts of the server device 10 while performing information processing related to the operation of the server device 10.

[0037] The input unit 105 includes one or more input interfaces. These input interfaces may be, for example, physical keys, capacitive keys, indicator devices, a touchscreen integrated with the display, or a microphone for receiving audio input. The input unit 105 accepts input of information used in the operation of the server device 10 and transmits the input information to the control unit 103.

[0038] The output unit 106 includes one or more output interfaces. These output interfaces may be, for example, a display or a speaker. The display may be, for example, an LCD (Liquid Crystal Display) or an OLED (Electro-Luminescence) display. The output unit 106 outputs information obtained through the operation of the server device 10.

[0039] The functions of server device 10 are implemented by a control program executed by a processor included in control unit 103. The control program is a program used to enable the computer to function as server device 10. Alternatively, some or all of the functions of server device 10 can also be implemented by dedicated circuitry included in control unit 103. Furthermore, the control program can also be stored on a non-temporary recording / storage medium that can be read by server device 10, and server device 10 reads from the medium.

[0040] The terminal device 12 includes a communication unit 111, a storage unit 112, a control unit 113, an input unit 115, an output unit 116, and a camera unit 117.

[0041] The communication unit 111 includes a communication module corresponding to wired or wireless LAN standards, and a module corresponding to mobile communication standards such as LTE, 4G, and 5G. The terminal device 12 connects to the network 11 via the communication unit 111, through a nearby router device or a mobile communication base station, and communicates with the server device 10 and the like via the network 11.

[0042] Storage unit 112 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of these. Semiconductor memories are, for example, RAM or ROM. RAM is, for example, SRAM or DRAM. ROM is, for example, EEPROM. Storage unit 112 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. Storage unit 112 stores information used in the operation of control unit 113 and information obtained through the operation of control unit 113.

[0043] The control unit 113 may have one or more general-purpose processors such as CPUs and MPUs (Micro Processing Units), or one or more dedicated processors such as GPUs that are specialized for specific processing. Alternatively, the control unit 113 may also have one or more dedicated circuits such as FPGAs and ASICs. The control unit 113 controls the operation of the terminal device 12 by operating according to a control / processing program or according to an operation process installed as a circuit. Furthermore, the control unit 113 sends and receives various information with the server device 10 and the like via the communication unit 111, and performs the operations of this embodiment.

[0044] The input unit 115 includes one or more input interfaces. These interfaces may include, for example, physical keys, capacitive keys, indicator devices, a touchscreen integrated with the display, and proximity sensors such as infrared sensors that detect user gestures. Additionally, the input interfaces may include microphones for receiving audio input and cameras for capturing video images. Furthermore, the input interfaces may also include scanners or cameras for scanning image codes, or IC card readers. The input unit 115 accepts input of information used in the operation of the control unit 113 and transmits the input information to the control unit 113.

[0045] The output unit 116 includes one or more output interfaces. These output interfaces may include, for example, a display and a speaker. The display may be, for example, an LCD or an OLED display. The output unit 116 outputs information obtained through the operation of the control unit 113.

[0046] The camera unit 117 includes: a camera that captures images of a subject based on visible light; and a range sensor that measures the distance to the subject to obtain a distance image. The camera, for example, captures images of the subject at 15-30 frames per second to generate a moving image composed of continuous video images. The range sensor includes a ToF (Time of Flight) camera, a LiDAR (Light Detection and Ranging) camera, or a stereo camera, generating a distance image of the subject that includes distance information. The camera unit 117 sends the video images and the distance image to the control unit 113.

[0047] The functions of the control unit 113 are implemented by executing a control program through a processor included in the control unit 113. The control program is a program used to enable the processor to function as the control unit 113. Alternatively, some or all of the functions of the control unit 113 can also be implemented through dedicated circuitry included in the control unit 113. Furthermore, the control program can also be stored on a non-temporary recording / storage medium that can be read by the terminal device 12, which reads the program from the medium.

[0048] In this embodiment, the control unit 113 acquires a camera image and a distance image of the caller via the camera unit 117, and collects the caller's voice using the microphone of the input unit 115. The control unit 113 encodes the camera image and distance image of the caller (used to generate a 3D model of the caller) and the voice information (used to reproduce the caller's voice) to generate encoded information. The control unit 113 may also perform arbitrary processing on the camera image (e.g., resolution change and cropping) during encoding. The control unit 113 sends the encoded information to other terminal devices 12 via the server device 10 through the communication unit 111. Additionally, the control unit 113 receives encoded information from other terminal devices 12 via the server device 10 through the communication unit 111. After decoding the encoded information received from other terminal devices 12, the control unit 113 uses the decoded information to generate a 3D model representing the caller using the other terminal device 12, and places this 3D model in virtual space. Furthermore, the control unit 113 can also use the camera image and distance image of the caller to generate a 3D model representing the caller and configure it in the virtual space. When generating the 3D model, the control unit 113 generates a polygon model using the distance image, performs texture mapping on the polygon model using the camera image, and thus generates the 3D model. However, the generation of the 3D model is not limited to the example shown here, and any method can be used. When the control unit 113 draws and generates a virtual space image for output, including a 3D model from a predetermined viewpoint within the virtual space, it displays the virtual space image through the output unit 116 and outputs the voice of the caller based on the caller's voice information. Through the actions of the control unit 113 and the like, the caller of the terminal device 12 can participate in virtual events and communicate with the caller in real time.

[0049] Figure 2 This is a sequence diagram illustrating the operational process of the virtual event providing system 1. The sequence diagram shows the process related to the communication between the server device 10 and multiple terminal devices 12 (referred to as terminal devices 12A and 12B for convenience). This process, for example, occurs when a caller using terminal device 12A, acting as both the administrator and participant of the virtual event, invites a caller using terminal device 12B to the virtual event. In the case of inviting multiple callers, the actions related to terminal device 12B shown here are performed by the individual terminal devices 12B of each caller, or by each of the multiple terminal devices 12B and the server device 10.

[0050] Figure 2The steps related to various information processing of the server device 10 and terminal device 12 are executed by their respective control units 103 and 113. Furthermore, the steps related to the transmission and reception of various information of the server device 10 and terminal device 12 are performed by the respective control units 103 and 113 respectively transmitting and receiving information to each other via communication units 101 and 111. In the server device 10 and terminal device 12, the respective control units 103 and 113 appropriately store the transmitted and received information in storage units 102 and 112, respectively. Furthermore, the control unit 113 of the terminal device 12 accepts various information inputs via the input unit 115 and outputs various information via the output unit 116.

[0051] In step S200, terminal device 12A receives input of setting information for a virtual event initiated by the caller. The setting information includes the virtual event schedule, seminar topic, and a list of participants. The list of participants includes participant names and email addresses. Then, in step S201, terminal device 12A sends the setting information to server device 10. Server device 10 receives the information from terminal device 12A. For example, terminal device 12A accesses a site provided by server device 10 for implementing virtual events to obtain an input screen for the setting information and displays the input screen to the user. Then, the setting information is sent to server device 10 by the caller inputting the setting information on the input screen.

[0052] In step S202, the server device 10 sets up a virtual event based on the setting information. The control unit 103 maps the information of the virtual event to the information of the participants and saves it to the storage unit 102.

[0053] In step S203, server device 10 sends authentication information to terminal device 12B. The authentication information is used to identify and authenticate the user of terminal device 12B, and includes information such as ID and password used when participating in virtual events. This information may be sent, for example, attached to an email address. Terminal device 12B receives the information sent from server device 10.

[0054] In step S205, terminal device 12B sends the authentication information and participation application information received from server device 10 to server device 10. The other party in the call operates terminal device 12B, using the authentication information sent from server device 10 to apply to participate in the virtual event. For example, terminal device 12B accesses a site provided by server device 10 for virtual events, obtains the authentication information and an input screen for the participation application information, and displays the input screen to the other party in the call. Furthermore, terminal device 12B accepts the information input by the other party in the call and sends it to server device 10.

[0055] In step S206, the server device 10 authenticates the other party to complete the participation in the call acceptance. In the storage unit 102, the identification information of the terminal device 12B and the identification information of the other party are matched and stored.

[0056] In steps S208 and S209, server device 10 sends an event start notification to terminal devices 12A and 12B, respectively. When terminal devices 12A and 12B receive the information from server device 10, they respectively begin collecting video and audio recordings of the caller and the other party in the call.

[0057] In step S210, the virtual event is implemented via server device 10 through terminal devices 12A and 12B. Terminal devices 12A and 12B respectively send and receive information for generating 3D models representing the caller and the other party, and information for generating voice recordings, via server device 10. Additionally, terminal devices 12A and 12B respectively output images of the virtual event, including 3D models representing themselves and the other party, and the other party's voice recordings to the caller and the other party, respectively.

[0058] Figure 3A , 3B This is a flowchart illustrating the operation process of terminal device 12 related to the implementation of virtual events. The process shown here is a common process for terminal devices 12A and 12B, and is described without distinguishing between terminal devices 12A and 12B.

[0059] Figure 3A The operation process of the control unit 113 when the terminal device 12 sends out information for generating a 3D model of the caller and information for the sound of the voice.

[0060] In step S302, the control unit 113 captures a visible light image of the participant at an arbitrarily set frame rate and acquires a distance image via the camera unit 117, and collects the voice of the speaker via the input unit 115. The control unit 113 acquires the visible light image and distance image from the camera unit 117, and acquires the sound information from the input unit 115.

[0061] In step S304, the control unit 113 encodes the camera image, distance image, and sound information to generate encoded information.

[0062] In step S306, the control unit 113 groups the encoded information through the communication unit 111 and sends it to the server device 10 for other terminal devices 12.

[0063] When the control unit 113 receives information corresponding to an operation performed by the caller to interrupt video / audio recording or to exit a virtual event (S308 "Yes"), it terminates the process. Figure 3A During the processing, if no information corresponding to the operation for interruption or exit is obtained ("No" in S308), steps S302 to S306 are executed to send information for generating a 3D model representing the caller and information for outputting sound to other terminal devices 12.

[0064] Figure 3B This pertains to the operation process of the control unit 113 when the terminal device 12 outputs an image including a virtual event of the other party in the call and the other party's voice. The control unit 113 receives data via the server device 10 from other terminal devices 12. Figure 3A When sending packets during the process, steps S310 to S313 are executed. When receiving information from multiple terminal devices 12 of the other party in the call, the control unit 113 executes steps S310 to S313 for each of the other terminal devices 12. Additionally, when the control unit 113 acquires the caller's camera image, distance image, and voice, steps S310 to S313 are executed.

[0065] In step S310, the control unit 113 decodes the encoded information contained in the packets received from the other terminal device 12 to obtain video images, distance images, and audio information. Additionally, when executing step S302, the control unit 113 obtains video images and distance images of the caller from the camera unit 117 and audio information from the input unit 115.

[0066] In step S312, the control unit 113 generates a 3D model representing each participant based on the camera image and the distance image.

[0067] In step S313, the control unit 113 configures 3D models representing the caller and the other party in the virtual space where the virtual event is held. The storage unit 112 pre-stores coordinate information of the virtual space and information indicating the coordinates of the 3D models of the caller and the other party to be configured according to, for example, the order of authentication.

[0068] In step S314, the control unit 113 draws and generates a virtual space image obtained by photographing multiple 3D models arranged in the virtual space from a virtual viewpoint.

[0069] In step S316, the control unit 113 displays a virtual space image and outputs sound through the output unit 116. That is, the control unit 113 outputs information about the image used to display an event with a 3D model configured in the virtual space to the output unit 116, and the output unit 116 outputs the virtual space image and the speaker's voice. For example, the output unit 116... Figure 4 As shown, a virtual space image 400 is displayed, including a caller 40 and call counterparts 41-46 represented by 3D models.

[0070] Control unit 113 repeatedly executes steps S310 to S316, allowing the caller to view an animation of a virtual space image including a 3D model of the other party while simultaneously hearing the other party's voice. Furthermore, control unit 113 can move the position of the caller 40 within the virtual space image 400 based on the caller's actions. These actions include, for example, dragging on a touch panel or using a pointing device, or gestures. Control unit 113 sends information related to the movement of the 3D model, such as the direction and amount of movement, to other terminal device 12. Thus, the caller 40 can be moved within the virtual space image 400 displayed towards the other party on other terminal device 12. Additionally, control unit 113 can also move the other parties 41 to 46 based on movement-related information sent from other terminal device 12.

[0071] Figure 5A , 5B This is a flowchart illustrating the operation process of terminal device 12 related to the execution of a group call in a virtual event. The process shown here is a common process for terminal devices 12A and 12B, and is described without distinguishing between terminal devices 12A and 12B.

[0072] Figure 5A This relates to the operation process in which the control unit 113 in the terminal device 12 performs processing for a caller to form a group consisting of desired call partners. For example, in response to an operation performed by the caller to instruct the formation of the group, the following steps are executed: Figure 5A The process.

[0073] In step S500, the control unit 113 receives a group formation instruction. The control unit 113 receives input from the input unit 115 corresponding to the operation performed by the caller. The caller selects a desired call partner in a virtual space image, for example, by tapping or clicking with an indicator device, and the instruction is given to form a group with the selected call partner.

[0074] In step S501, the control unit 113 notifies that a group has been formed. The control unit 113 stores the information of the caller's selected counterparty, and notifies the terminal devices 12 of the callers within the group and the terminal devices 12 of other callers outside the group of the group's formation. The notification includes information about the callers and counterparties within the group. As a condition for the control unit 113 to proceed to the next step, it may also be conditional upon obtaining consent from the terminal devices 12 of the callers within the group to join the group.

[0075] In step S502, the control unit 113 displays the formed group. The control unit 113 displays the callers and their counterparts within the group in a different style than other callers outside the group. For example, in Figure 6AIn the virtual space image 400 shown, the callers 40, 41, and 42 within the group are displayed in a style different from the other callers 43 to 46 outside the group. The different display styles include the addition of effects such as glows and shadows, emphasis on outlines, increased polygon count, and enlarged size. Alternatively, the control unit 113 can bring the callers 40 and 41 and 42 within the group closer together and surround them with a frame 60. However, as long as the callers 40 and 41 and 42 within the group are distinguishable from the other callers 43 to 46, the display style is not limited to that shown here.

[0076] In step S503, the control unit 113 initiates a group call. The control unit 113 outputs the voices of other participants within the group in a style different from those of other participants outside the group. For example, in... Figure 6A In the example of the virtual space image 400, the control unit 113 outputs the voices of the callers 41 and 42 within the group at a higher volume than the voices of the other callers 43-46 outside the group. Alternatively, the control unit 113 can also mute the voices of the other callers 43-46 outside the group. In this case, the control unit 113 can also display a string corresponding to the voices of the other callers 43-46 outside the group. This makes it easier for callers to focus on the group call.

[0077] Figure 5B This relates to the process in which the control unit 113 in terminal device 12 performs actions for a caller to participate in a group formed by the other party in the call. For example, this is performed in other terminal devices. Figure 5A In step S501, when a notification of group formation is received, the control unit 113 executes the procedure in response to the notification. Figure 5B The process.

[0078] In step S504, the control unit 113 receives a notification of group formation. Based on the notification, the control unit 113 identifies the callers included in the group. Here, the control unit 113 may also output a message to the caller confirming whether they agree to join the group, accept input indicating agreement or disagreement, and return this information to other terminal devices 12.

[0079] In step S505, the control unit 113 joins the group. The control unit 113 determines the call partners within the group and saves the information of the determined call partners by matching them together.

[0080] In step S506, the control unit 113 displays the participating groups. For example, the control unit 113... Figure 6A The example shown displays the callers and their counterparts within the group in a different style than other callers outside the group.

[0081] In step S508, the control unit 113 initiates a group call. The control unit 113 outputs the voices of the callers within the group in a different style than those of the callers outside the group.

[0082] By executing the above-described procedure in each terminal device 12 Figure 5A , Figure 5B The process involves multiple terminal devices 12 performing a group call. Furthermore, terminal devices 12 outside the group display the group members in a different style from the other callers via notifications from the terminal devices 12 of the callers forming the group. Thus, callers outside the group can recognize the formation of the group.

[0083] Even if the caller selects a caller from a group they are not currently in, the call will still be executed. Figure 5A The process. For example, the control unit 113 displays as follows: Figure 6B The virtual space image 400 is shown. Here, the state of a group formed by callers 41 and 42, in which caller 40 is not present, is shown. By surrounding callers 41 and 42 with borders 61 and displaying them in a style different from other callers 43-46, it is shown that a group has been formed. The control unit 113 accepts input from the caller, such as tapping or using gestures, to select callers 41 and 42. Alternatively, the caller can also select within the area of ​​borders 61 by tapping or using gestures. In this case, in step S501, the control unit 113 stores the information of the callers selected by the caller in correspondence and notifies the terminal devices 12 of the callers within the group and the terminal devices 12 of other callers outside the group of the group update. Thus, the caller can join an existing group to perform a group call.

[0084] Additionally, if the caller is already in a group and selects a caller outside the group, this also applies. Figure 5A The process involves adding new callers to an existing group. In this case, in step S501, the control unit 113 stores the information of the callers' existing group partners and the information of the newly selected callers, and notifies the terminal devices 12 of the callers within the group and the terminal devices 12 of other callers outside the group of the group update. Thus, the caller can add new callers to an existing group to perform a group call.

[0085] The caller is also executed when the other party in the call is selected from those who have joined the group. Figure 5B The control unit 113 stores information about the callers identified in the group from the notification generated by the group. This allows callers to join groups they have already joined to conduct group calls.

[0086] Additionally, this also applies when a caller is selected from outside the group if the caller is already in the group. Figure 5BThe process is as follows. For example, in step S505, the control unit 113 updates the group and joins. The control unit 113 determines the caller in the group and the new caller, and saves the information of the determined caller by matching them together. Thus, the caller can update the groups they have joined and join the updated group to perform a group call.

[0087] In this embodiment, the terminal device 12 is further capable of enabling a caller to conduct group calls in multiple other groups. This is achieved through the terminal device 12. Figure 5A as well as Figure 5B During a group call, the caller instructs the terminal device 12 to create or join other groups with the other callers. The terminal device 12 then executes commands targeting these other callers or groups. Figure 5A as well as Figure 5B The process.

[0088] Figure 7A This illustrates an example of a virtual space image 400 in which a caller forms or joins another group during a group call. Here, it shows the state where a caller 40, while conducting a group call with callees 41 and 42 within a group enclosed by border 60, forms another group with callees 43 and 44, or joins a group of callees 43 and 44. In this case, instead of updating the group including callees 41 and 42 to include callees 43 and 44, the caller performs an operation to select to join different groups simultaneously. For example, the caller performs tapping, gestures, or other operations to select relevant functions from a pop-up menu. When the caller selects callees 43 and 44 or their group, the control unit 113 displays a clone 40a of the caller and forms and displays a group consisting of the clone 40a and callees 43 and 44. Here, the group consisting of clone 40a and call partners 43 and 44 is displayed in a different style than call partners 45 and 46, which are not included in any group. The different style is indicated by being surrounded by a frame 70. Furthermore, on the terminal devices 12 of each call partner 43 and 44, clones 40a of the callers participating in the group are displayed, and each call partner can recognize the caller's participation.

[0089] In this embodiment, when a caller participates in multiple groups simultaneously, the terminal device 12 outputs the audio of the multiple group calls to the caller. However, the terminal device 12 operates in a manner that selectively transmits the caller's audio information to the terminal devices 12 of the primary group among the multiple groups, but not to the terminal devices 12 of the other parties in other secondary groups. For example, in Figure 7AIn the example, when a caller uses groups 41 and 42 as the primary group and groups 43 and 44 as secondary groups for a group call, the control unit 113 sends the caller's voice information to the respective terminal devices 12 of groups 41 and 42, and receives the voice information of groups 41 and 42 from their respective terminal devices 12. Alternatively, the control unit 113 may not send the caller's voice information to the respective terminal devices 12 of groups 43 and 44, but instead receive the voice information of groups 43 and 44 from their respective terminal devices 12. This allows the caller to understand the content of the calls in multiple groups while avoiding conversation confusion. Furthermore, in group calls within secondary groups, the control unit 113 can also display a string corresponding to the voice instead of outputting the voice of the other party.

[0090] In addition, such as Figure 7A As shown, the control unit 113 displays the clone 40a in a different style than the caller 40. This makes it easier for the caller to distinguish the primary and secondary groups, for example, when the group the caller 40 participates in is considered the primary group and the group the clone 40a participates in is considered the secondary group. Furthermore, the control unit 113 can also display the callers 41 and 42 of the primary group in a different style than the callers 43 and 44 of the secondary group, for example, through lighting effects, shadows, and contour coordination.

[0091] Furthermore, the control unit 113 can appropriately change the primary group based on the operation performed by the caller. Such operations include, for example, tapping or gesture operations to select the primary group from the pop-up menu.

[0092] Figure 7B It is displayed in Figure 7A This is an example of a virtual space image 400 on the terminal device 12 of the other party 44 in a group call. Here, for the group of other parties 44, the clone 40a of the caller 40 and the other party 44 are displayed. On the other hand, the group of caller 40, the other parties 41 and 42 for the caller 40 are also displayed. In this way, the existence of the clone 40a can also be seen on the other party 44 side, so the sense of incongruity in the group call is reduced.

[0093] In the above description, the caller and the other party are represented by a 3D model based on camera images, but it can also be an image of a virtual avatar, character, etc., generated without camera images. In this case, terminal device 12 sends the information of the image used to construct the virtual avatar, character, etc., to other terminal devices 12 instead of camera images, distance images, etc. Thus, the images of the caller and the other party can be displayed on each terminal device 12.

[0094] According to the above implementation method, callers can participate in multiple groups for group calls, thus improving the convenience for users making calls in virtual space.

[0095] In the foregoing, embodiments have been described with reference to the accompanying drawings and examples. However, it is important to note that those skilled in the art can readily make various modifications and alterations based on this disclosure. Therefore, it is desirable to understand that these modifications and alterations are included within the scope of this disclosure. For example, the functions included in each unit, step, etc., can be reconfigured in a logically consistent manner, and multiple units, steps, etc., can be combined into one or divided.

Claims

1. A terminal device comprising: Department of Communications; Output section; and The control unit communicates with the terminal devices of multiple call partners through the communication unit, and outputs an image of a virtual space including the images of the multiple call partners and their audio through the output unit. The control unit outputs the images or sounds of the callers from multiple groups to which the caller belongs in a different style from the other callers, and sends information to the terminal devices of the multiple groups for outputting the caller's images or sounds in a different style from the other callers' terminal devices. The multiple groups include primary groups and secondary groups. When the caller is simultaneously participating in both the primary group and the secondary group. The control unit sends the caller's voice information to the terminal device of the other party in the main group, and receives the voice information of the other party in the main group from the terminal device of the other party in the main group. The control unit does not send the caller's voice information to the terminal device of the other party in the secondary group, but receives the voice information of the other party in the secondary group from the terminal device of the other party in the secondary group.

2. The terminal device according to claim 1, wherein, The control unit selects the multiple groups from the multiple callers based on the operation performed by the caller.

3. The terminal device according to claim 1 or 2, wherein, The control unit then outputs the image or sound of the other party in the first group of calls in a different style than that of the other party in the second group of calls.

4. The terminal device according to claim 3, wherein, The control unit outputs a string corresponding to the voice of the other party in the second group of calls.

5. A computer-readable, non-transitory medium for storing programs. The program is executed by the control unit of the terminal device, which has a communication unit, an output unit, and a control unit. The program causes the control unit to perform the steps of communicating with the terminal devices of multiple call partners through the communication unit and causing the output unit to output an image of a virtual space including images of the multiple call partners and their audio. in, The program causes the control unit to further execute the step of outputting the image or sound of the caller in a different style from that of the other callers in multiple groups, and sending information for outputting the caller's image or sound in a different style from that of the other callers' terminal devices to the multiple groups of callers. The multiple groups include primary groups and secondary groups. When the caller is simultaneously participating in the main group and the secondary group, the program causes the control unit to further perform the following: sending the caller's voice information to the terminal device of the caller in the main group and receiving the voice information of the caller in the main group from the terminal device of the caller in the main group; and not sending the caller's voice information to the terminal device of the caller in the secondary group and receiving the voice information of the caller in the secondary group from the terminal device of the caller in the secondary group.

6. The computer-readable, non-transitory medium for storing the program according to claim 5, wherein, The program causes the control unit to select the multiple groups from the multiple callers based on the actions performed by the caller.

7. The computer-readable, non-transitory medium for storing the program according to claim 5 or 6, wherein, The program causes the control unit to output the image or sound of the other party in the first group of calls in a different style than that of the other party in the second group of calls.

8. The computer-readable, non-transitory medium for storing the program according to claim 7, wherein, The program causes the control unit to output a string corresponding to the voice of the other party in the second group of calls.

9. An operating method for a terminal device having a communication unit, an output unit, and a control unit, comprising: The control unit communicates with the terminal devices of multiple call partners through the communication unit, and outputs an image of a virtual space including the images of the multiple call partners and the voices of the multiple call partners through the output unit; as well as The control unit then outputs the image or sound of the other party in multiple groups to which the caller belongs in a different style from the other callers, and sends information to the terminal devices of the multiple groups for outputting the caller's image or sound in a different style from the other callers' terminal devices. The multiple groups include primary groups and secondary groups. When the caller is simultaneously participating in both the primary group and the secondary group. The control unit sends the caller's voice information to the terminal device of the other party in the main group, and receives the voice information of the other party in the main group from the terminal device of the other party in the main group. The control unit does not send the caller's voice information to the terminal device of the other party in the secondary group, but receives the voice information of the other party in the secondary group from the terminal device of the other party in the secondary group.

10. The method of action according to claim 9, wherein, The control unit selects the multiple groups from the multiple callers based on the operation performed by the caller.

11. The method of action according to claim 9 or 10, wherein, The control unit outputs the image or sound of the other party in the first group of calls in a different style than that of the other party in the second group of calls.

12. The method of action according to claim 11, wherein, The control unit outputs a string corresponding to the voice of the other party in the second group of calls.

Images