Terminal device, operation method of terminal device, and system

Abstract

The present disclosure relates to a terminal device, a method of operating the terminal device, and a system. It facilitates the participation of participants in activities in a virtual space. The terminal device has a communication unit and a control unit that communicates through the communication unit. The control unit receives information from another terminal device for generating a 3D model of a participant who uses the other terminal device to participate in an activity in the virtual space, generates the 3D model in a form corresponding to the role of the participant in the activity, and outputs information for displaying an image obtained by rendering the virtual space in which the 3D model is arranged.

Description

Technical Field

[0001] The present disclosure relates to a terminal device, an operation method of the terminal device, and a system. Background Art

[0002] It is known that computers at multiple locations communicate via a network and a method of holding a meeting in a virtual space on the network. Various techniques for improving the convenience of participants in such a network meeting have been proposed. For example, Patent Document 1 discloses the following system: In the images of meeting participants displayed on each computer, the images of participants with the right to speak are distinguished from the images of other participants.

[0003] Prior Art Documents

[0004] Patent Documents

[0005] Patent Document 1: Japanese Patent Laid-Open No. 8-331534 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] There is room for further improving the convenience of participants in participating in activities in a virtual space on the network.

[0008] The present disclosure provides a terminal device and the like that contribute to the convenience of participants in participating in activities in a virtual space.

[0009] Solutions to the Problems

[0010] The terminal device in the present disclosure includes a communication unit and a control unit that communicates through the communication unit. The control unit receives information for generating a 3D model of a participant who uses the other terminal device to participate in an activity in a virtual space from another terminal device, generates the 3D model in a form corresponding to the role of the participant in the activity, and outputs information for displaying an image obtained by rendering the virtual space in which the 3D model is arranged.

[0011] The operation method of the terminal device in the present disclosure is an operation method of a terminal device having a communication unit and a control unit that communicates through the communication unit. The control unit sends information for generating a 3D model of a participant who participates in an activity in a virtual space to another terminal device, generates the 3D model in a form corresponding to the role of the participant in the activity by the other terminal device, and outputs information for displaying an image obtained by rendering the virtual space in which the 3D model is arranged.

[0012] The system in this disclosure is a system having a plurality of terminal devices that communicate via a server device. Among them, the first terminal device sends information for generating a 3D model representing a participant who uses the first terminal device to participate in an activity in a virtual space to the second terminal device, and the second terminal device generates the 3D model in a form corresponding to the role of the participant in the activity, and outputs information for displaying an image obtained by rendering the virtual space in which the 3D model is configured.

[0013] Effects of the Invention

[0014] According to the terminal device and the like in this disclosure, it is possible to contribute to the convenience of participants participating in activities in a virtual space. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a diagram showing a configuration example of a virtual activity providing system.

[0016] Figure 2 It is a sequence diagram showing an operation example of a virtual activity providing system.

[0017] Figure 3 It is a diagram showing an example of an input screen for setting information.

[0018] Figure 4 It is a diagram showing an example of setting information.

[0019] Figure 5A It is a flowchart showing an operation example of a terminal device.

[0020] Figure 5B It is a flowchart showing an operation example of a terminal device.

[0021] Figure 6A It is a diagram showing an example of the form of a 3D model.

[0022] Figure 6B It is a diagram showing an example of the form of a 3D model.

[0023] Figure 7A It is a diagram showing an example of a virtual space image.

[0024] Figure 7B It is a diagram showing an example of a virtual space image.

[0025] Figure 7C It is a diagram showing an example of a virtual space image.

[0026] Figure 8A It is a flowchart showing an operation example of a terminal device in a modified example.

[0027] Figure 8B It is a flowchart showing an operation example of a terminal device in a modified example.

[0028] Figure 9 This is a diagram showing an example of a virtual space image in a modified example.

[0029] Description of Reference Numerals

[0030] 1. Virtual event providing system; 10. Server device; 11. Network; 12. Terminal device; 101, 111. Communication unit; 102, 112. Storage unit; 103, 113. Control unit; 105, 115. Input unit; 106, 116. Output unit; 117. Imaging unit. Detailed Description of the Embodiment

[0031] Hereinafter, the embodiment will be described.

[0032] Figure 1 This is a diagram showing a configuration example of the virtual event providing system 1 in an embodiment. The virtual event providing system 1 is a system for providing an event in a virtual space that a user can participate in using a terminal device, that is, a virtual event. A virtual event is an event in which multiple participants transmit information through speech or the like in a virtual space, and each participant is represented by a 3D model of their own. The virtual event providing system 1 includes a server device 10 and a plurality of terminal devices 12 connected via a network 11 so as to be able to communicate with each other.

[0033] The server device 10 is, for example, a server computer belonging to a cloud computing system or other computing system and functioning as a server installed with various functions. The server device 10 may also be composed of two or more server computers connected in a manner capable of communicating information and operating in cooperation. The server device 10 performs the transmission and reception of information and information processing required for providing a virtual event.

[0034] The terminal device 12 is an information processing device having a communication function and is used by a user (hereinafter referred to as a participant) who participates in a virtual event provided by the server device 10. The terminal device 12 is, for example, an information processing terminal such as a smartphone or a tablet terminal, or an information processing device such as a personal computer.

[0035] The network 11 is, for example, the Internet, but includes an ad hoc network, a LAN (Local Area Network), a MAN (Metropolitan Area Network), or other networks or any combination of them.

[0036] In this embodiment, the terminal device 12 includes a communication unit 111 and a control unit 113. The control unit 113 receives, from the terminal device 12, information for generating a 3D model of a participant who uses another terminal device 12 to participate in an activity in a virtual space, generates the 3D model in a form corresponding to the importance of the role of the participant in the activity, and outputs information for displaying an image obtained by rendering the virtual space in which the 3D model is arranged. By changing the form of the 3D model representing the participant according to the importance of the role of the participant, each participant in the virtual activity can visually recognize the roles of other participants, and it is easy to focus attention on the participants with high importance roles. Therefore, it can contribute to the convenience of the participants.

[0037] The structures of the server device 10 and the terminal device 12 will be described in detail.

[0038] The server device 10 includes a communication unit 101, a storage unit 102, a control unit 103, an input unit 105, and an output unit 106. In the case where the server device 10 is composed of two or more server computers, these structures are appropriately arranged in the two or more computers.

[0039] The communication unit 101 includes one or more communication interfaces. The communication interface is, for example, a LAN interface. The communication unit 101 receives information for the operation of the server device 10 and also transmits information obtained by the operation of the server device 10. The server device 10 is connected to the network 11 through the communication unit 101 and communicates with the terminal device 12 via the network 11.

[0040] The storage unit 102 includes, for example, one or more semiconductor memories that function as a main storage device, an auxiliary storage device, or a cache memory, one or more magnetic memories, one or more optical memories, or a combination of at least two of them. The semiconductor memory is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The RAM is, for example, an SRAM (Static RAM) or a DRAM (Dynamic RAM). The ROM is, for example, an EEPROM (Electrically Erasable Programmable ROM). The storage unit 102 stores information for the operation of the server device 10 and information obtained by the operation of the server device 10.

[0041] The control unit 103 includes one or more processors, one or more dedicated circuits, or a combination thereof. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit), or a dedicated processor such as a GPU (Graphics Processing Unit) dedicated to specific processing. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 103 controls each part of the server device 10 while performing information processing related to the operation of the server device 10.

[0042] The input unit 105 includes one or more input interfaces. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen integrated with the display, or a microphone that accepts voice input. The input unit 105 accepts an operation of inputting information for the operation of the server device 10 and sends the input information to the control unit 103.

[0043] The output unit 106 includes one or more output interfaces. The output interface is, for example, a display or a speaker. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The output unit 106 outputs the information obtained through the operation of the server device 10.

[0044] The functions of the server device 10 are realized by the processor included in the control unit 103 executing a control program. The control program is a program for causing a computer to function as the server device 10. In addition, some or all of the functions of the server device 10 can also be realized by the dedicated circuit included in the control unit 103. In addition, the control program can also be stored in a non-volatile recording and storage medium readable by the server device 10, and the server device 10 reads it from the medium.

[0045] The terminal device 12 has a communication unit 111, a storage unit 112, a control unit 113, an input unit 115, an output unit 116, and a camera unit 117.

[0046] The communication unit 111 has a communication module corresponding to the wired or wireless LAN standard, a module corresponding to mobile communication standards such as LTE, 4G, 5G, etc. The terminal device 12 is connected to the network 11 through the communication unit 111 via a nearby routing device or a mobile communication base station, and performs information communication with the server device 10 etc. via the network 11.

[0047] The storage unit 112 includes one or more semiconductor memories, one or more magnetic memories, one or more optical memories, or a combination of at least two of them. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The storage unit 112 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 112 stores information for controlling the operation of the control unit 113 and information obtained through the operation of the control unit 113.

[0048] The control unit 113 has, for example, one or more general-purpose processors such as a CPU or an MPU (Micro Processing Unit), or one or more dedicated processors such as a GPU dedicated to specific processing. Alternatively, the control unit 113 may have one or more dedicated circuits such as an FPGA or an ASIC. The control unit 113 operates according to a control and processing program or according to an operation procedure installed as a circuit, thereby uniformly controlling the operation of the terminal device 12. Further, the control unit 113 transmits and receives various information to and from the server device 10 etc. via the communication unit 111, and executes the operation of the present embodiment.

[0049] The input unit 115 includes one or more input interfaces. The input interface includes, for example, physical keys, capacitive keys, pointing devices, and a touch screen integrally provided with a display. In addition, the input interface includes a microphone for accepting voice input and a camera for taking in captured images. Further, the input interface may also include a scanner or a camera for scanning image codes, an IC card reader, etc. The input unit 115 accepts an operation of inputting information for controlling the operation of the control unit 113, and sends the input information to the control unit 113.

[0050] The output unit 116 includes one or more output interfaces. The output interface includes, for example, a display and a speaker. The display is, for example, an LCD or an organic EL display. The output unit 116 outputs information obtained through the operation of the control unit 113.

[0051] The imaging unit 117 includes a camera that captures a captured image of a subject based on visible light and a distance measurement sensor that measures the distance to the subject and obtains a distance image. The camera captures the subject at, for example, 15 to 30 frames per second to generate a moving image composed of consecutive captured images. The distance measurement sensor includes a ToF (Time Of Flight) camera, LiDAR (Light Detection And Ranging), a stereo camera, and generates an image of the subject including distance information. The imaging unit 117 sends the captured image and the distance image to the control unit 113.

[0052] The functions of the control unit 113 are implemented by a processor included in the control unit 113 executing a control program. The control program is a program for causing the processor to function as the control unit 113. In addition, part or all of the functions of the control unit 113 can also be implemented by a dedicated circuit included in the control unit 113. Further, the control program can also be stored in a non-volatile recording and storage medium readable by the terminal device 12, and the terminal device 12 reads it from the medium.

[0053] In the present embodiment, the control unit 113 acquires a captured image and a distance image of a user of the terminal device 12, i.e., a participant (hereinafter referred to as this participant), through the imaging unit 117, and collects the speech sound of this participant through the microphone of the input unit 115. If the captured image and distance image of this participant for generating a 3D model of this participant and the sound information for reproducing the sound of this participant are encoded to generate encoded information, the control unit 113 transmits the encoded information to other terminal devices 12 via the server device 10 through the communication unit 111. At the time of encoding, the control unit 113 can also perform arbitrary processing (such as resolution change and cropping, etc.) on the captured image and the like. In addition, if the control unit 113 receives the encoded information transmitted from other terminal devices 12 through the communication unit 111 via the server device 10, the control unit 113 decodes the encoded information. Then, the control unit 113 uses the decoded information to generate a 3D model representing another participant (hereinafter referred to as another participant) using other terminal devices 12, and arranges the 3D model in a virtual space. The control unit 113 can also generate a 3D model of this participant and arrange it in the virtual space. When generating a 3D model, the control unit 113 uses the distance image of another participant to generate a polygon model, and performs texture mapping using the captured image of another participant on the polygon model, thereby generating a 3D model of another participant. However, the generation of the 3D model is not limited to this example, and any method can be adopted. When generating a virtual space image for output including the 3D model captured from a predetermined viewpoint in the virtual space, the control unit 113 uses the output unit 116 to display the virtual space image and outputs sound. Through the actions of the control unit 113 and the like in this way, this participant of the terminal device 12 can participate in the virtual event in real time and have a conversation with other participants.

[0054] Figure 2This is a sequence diagram for explaining the operation steps of the virtual event providing system 1. This sequence diagram shows the steps involved in the associated operations of the server device 10 and multiple terminal devices 12 (when distinguished separately, referred to as terminal devices 12A and 12B). The terminal device 12A is used by the administrator and participant of the virtual event (hereinafter referred to as participant A). The terminal device 12B is used by participants other than the administrator (hereinafter referred to as participant B). Here, the operation steps performed by one terminal device 12B are shown, but these operation steps can also be executed by each of the multiple terminal devices 12B.

[0055] Figure 2 The steps involved in various information processing of the server device 10 and the terminal device 12 in are executed by their respective control units 103 and 113. In addition, the steps involved in the transmission and reception of various information between the server device 10 and the terminal device 12 are executed by the respective control units 103 and 113 to mutually transmit and receive information via the communication units 101 and 111 respectively. In the server device 10 and the terminal device 12, the control units 103 and 113 respectively store the transmitted and received information appropriately in the storage units 102 and 112. Furthermore, the control unit 113 of the terminal device 12 accepts the input of various information through the input unit 115 and outputs various information through the output unit 116.

[0056] In step S200, the terminal device 12A accepts the input of the setting information of the virtual event performed by participant A. The setting information includes the type of the virtual event and the roles of the participants in the virtual event. Then, in step S201, the terminal device 12A sends the setting information to the server device 10. The server device 10 receives the information sent from the terminal device 12A. For example, the terminal device 12A accesses the website provided by the server device 10 for implementing the virtual event to obtain the input screen of the setting information, and displays the input screen to participant A. Then, by participant A inputting the setting information in the input screen, the setting information is sent to the server device 10.

[0057] Figure 3 Shows an example of the input screen of the setting information. The input screen 30 has an input field 31 for selecting the activity category, an input field 32 for the email address of inviting other participants to the activity, and an input field 33 for the roles of each participant in the activity. The input fields 31 and 33 display options, for example, to prompt participant A to make a selection. Participant A selects the desired value from the activity categories 41. In addition, participant A inputs the email address of participant B who is invited to the activity into the input field 32 and selects the role assigned to participant B from the corresponding input field 33. Furthermore, participant A selects their own role in the input field 33. In addition, as shown here, each role can also be associated with an arbitrarily pre-set importance level.

[0058] Figure 4 This table 40 shows an example of an activity type, a role, and the importance of the role. The table 40 includes an activity type 41, a role 42, and an importance 43 as information items. The server device 10 stores such a table 40 in the storage unit 102 in advance.

[0059] Activity category 41 has values ​​such as "discussion", "demonstration", and "entertainment". The classification of activity category 41 is an example of classification based on the way of information exchange between participants of virtual activities. For example, "discussion" includes various discussions and debates in administration, justice, and business, which are ways of exchanging opinions and information with relatively clear opposing structures. "Demonstration" includes ways of transmitting information from one or a relatively small number of senders to a relatively large audience, such as demonstrations in business, lectures in business and education, handouts, academic presentations, and performances in music and art. "Entertainment" includes relatively irregular ways of exchanging opinions and information between participants, such as seminars and social gatherings. The values ​​of activity category 41 are illustrative and may be values ​​other than those shown here.

[0060] The value of the role 42 is different for each of the values ​​of "discussion", "demonstration", and "entertainment" of the activity category 41. And, a correspondence is established between the value of the importance 43 and each value of the role 42. For example, the value of the role 42 corresponding to "discussion" is "chairman", "team representative", and "team member", and the corresponding importance 43 is "medium", "high", and "low", respectively. In addition, the value of the role 42 corresponding to "demonstration" is "presenter", "MC", and "audience", and the corresponding importance 43 is "high", "medium", and "low", respectively. And, the value of the role 42 corresponding to "entertainment" is "organizer", "assistant", and "guest", and the corresponding importance 43 is "high", "low", and "medium", respectively. The combination of the value of the role 42 and the value of the importance 43 is an example, and combinations other than those shown here are also possible.

[0061] The participant A selects appropriate values ​​from the activity category 41 , the role 42 , and the importance 43 in the table 40 on the input screen 30 , and inputs setting information.

[0062] return Figure 2 In step S202, the server device 10 sets a virtual event based on the setting information and sets roles for scheduled participants in the virtual event.

[0063] In step S203, the server device 10 sends authentication information to the terminal device 12B. The authentication information is information for identifying and authenticating the participant B using the terminal device 12B, such as the ID, password, etc. used when participating in the virtual event. This information is sent, for example, attached to an email. The terminal device 12B receives the information sent from the server device 10.

[0064] In step S205, the terminal device 12B sends the authentication information received from the server device 10 and the information for the participation application to the server device 10. The participant B operates the terminal device 12B and uses the authentication information sent from the server device 10 to apply to participate in the virtual event. For example, the terminal device 12B accesses the website for the virtual event provided by the server device 10, obtains the input screens for the authentication information and the information for the participation application, and displays the input screens to the participant B. Then, the terminal device 12B accepts the information respectively input by the participant B and sends it to the server device 10.

[0065] In step S206, the server device 10 authenticates the participant B and sets the role, completing the acceptance of participation. The identification information of the terminal device 12B, the identification information of the participant B, and the information on the role and importance are stored in the storage unit 102 in a corresponding relationship.

[0066] In steps S208 and S209, the server device 10 sends the start notification of the event and the role information to the terminal devices 12A and 12B respectively. The role information includes the information on the respective roles and importance of the participants A and B. When the terminal devices 12A and 12B receive the information sent from the server device 10 and store it in the storage unit 112 respectively, they start collecting the images and speaking voices of the participants A and B respectively.

[0067] In step S210, the virtual event is implemented by the terminal devices 12A and 12B via the server device 10. The terminal devices 12A and 12B respectively send and receive, via the server device 10, the information for generating the 3D models of the participants A and B and the information of the speaking voices. In addition, the terminal devices 12A and 12B respectively output the images of the virtual event including the 3D models of the other participants and the speaking voices of the other participants to the participants A and B.

[0068] Figure 5A 、 5B is a flowchart showing the action steps of the terminal device 12 involved in the implementation of the virtual event.

[0069] Figure 5A relates to the action steps of the control unit 113 when each terminal device 12 sends the information for generating the 3D model of the participant using that terminal device 12.

[0070] In step S501, the control unit 113 acquires character information. The control unit 113 acquires the character information received from the server device 10 and stored in the storage unit 112, and acquires information on the character of the participant and its importance.

[0071] In step S502, the control unit 113 determines the form of the 3D model representing the participant. The control unit 113 determines the form of the 3D model according to the importance of the character of the participant.

[0072] For example, the control unit 113 determines the number of polygons used to generate the 3D model of the participant according to the importance of the character of the participant. Specifically, as the importance of the character decreases, the control unit 113 reduces the number of polygons used to generate the 3D model. For example, information on the number of polygons at three stages that decreases successively corresponding to the character importance levels of "high", "medium", and "low" is stored in the storage unit 112 in advance, and the control unit 113 reads the number of polygons corresponding to the importance of the character of the participant. It should be noted that the control unit 113 can also dynamically adjust the number of polygons within the range of the number of polygons set according to the importance level of the character according to the processing load and communication speed.

[0073] In addition, for example, the control unit 113 determines the thickness of the 3D model of the participant in a predetermined axial direction according to the importance of the character of the participant. Specifically, the control unit 113 makes the thickness of the participant in the optical axis direction during shooting thinner as the importance of the character decreases. For example, information on the thickness at three stages that decreases successively corresponding to the character importance levels of "high", "medium", and "low" is stored in the storage unit 112 in advance, and the control unit 113 reads the thickness corresponding to the importance of the character of each participant.

[0074] In step S503, the control unit 113 captures the visible light image of the participant and acquires the distance image at an arbitrarily set frame rate through the imaging unit 117, and collects the speech sound of the participant through the input unit 115. The control unit 113 acquires the imaging image based on visible light and the distance image from the imaging unit 117, and acquires the sound information from the input unit 115.

[0075] In step S504, the control unit 113 generates 3D model information required to generate the 3D model of the participant in the form determined in step S502 based on the imaging image and the distance image. The 3D model information is generated by selecting, from the imaging image and the distance image, the information used to construct the 3D model with the determined number of polygons or thickness.

[0076] In step S505, the control unit 113 encodes the 3D model information and the sound information to generate encoded information.

[0077] In step S506, the control unit 113 packs the encoded information via the communication unit 111 and sends it to the server device 10.

[0078] If the control unit 113 acquires the information input corresponding to the operation for interrupting shooting, collecting sound, or the operation for exiting the virtual event (Yes in S508), it ends Figure 5A the processing steps. During the period when the information corresponding to the operation for interruption or exit is not acquired (No in S508), steps S503 to S506 are executed. In this way, the terminal device 12 sends the information for generating the 3D model representing the participant to other terminal devices 12.

[0079] Figure 5B It relates to the operation steps of the control unit 113 when the terminal device 12 outputs an image of the virtual event including the 3D models of other participants and the voices of other participants based on the information received from other terminal devices 12. Figure 5B The step is executed when the control unit 113 receives, via the server device 10, the packet of the moving image information sent by other terminal devices 12 by executing Figure 5A the step.

[0080] In step S510, the control unit 113 decodes the encoded information included in the packet and acquires the 3D model information and the sound information.

[0081] In step S511, the control unit 113 generates the 3D models representing other participants based on the 3D model information. For example, it generates the 3D models 60, 61, or 62 schematically shown in Figure 6A . For example, when the importance of the role of other participants is "high", the 3D model 60 is represented by many polygons to show a fine curved surface. When the importance is "medium", the 3D model 61 is generated from fewer polygons, and when the importance is "low", the 3D model 62 is generated from fewer polygons. Therefore, as the number of polygons decreases, the flat parts become prominent. That is, the higher the importance of the role of other participants, the more realistic the generated 3D model. Or, it can also be generated as in Figure 6B3D model 60, 63 or 64 schematically shown in the figure. For example, the 3D model 60 in the case where the importance of other participants is "high" is represented by the thickness in the optical axis direction 65 corresponding to the distance image. The 3D model 63 in the case where the importance is "medium" is represented by a thinner thickness in the optical axis direction 65, and the 3D model 64 in the case where the importance is "low" is represented by a further thinner thickness in the optical axis direction 65. The control unit 113 may also increase the transparency of the 3D model as the thickness is thinner. By doing so, as the importance becomes lower, the 3D model becomes thinner and the transparency increases. In this way, the higher the importance of the role of the participant, the more realistic and more eye-catching 3D model is generated.

[0082] In step S512, the control unit 113 generates 3D models representing other participants and arranges them in the virtual space where the virtual event is held. The storage unit 112 stores in advance the coordinate information of the virtual space and the information of the coordinates at which the 3D model should be arranged according to the role of the participant for each activity type. The control unit 113 selects the virtual space corresponding to the activity type specified in the setting information. The control unit 113 arranges the generated 3D model at the coordinates in the virtual space corresponding to the role of the other participant.

[0083] In step S513, the control unit 113 renders and generates a virtual space image obtained by photographing the 3D model arranged in the virtual space from a virtual viewpoint. The virtual viewpoint is the viewpoint of the participant.

[0084] Alternatively, the control unit 113 may generate a 3D model representing the participant in step S512 , and render a virtual space image captured from a virtual viewpoint including the 3D model of the participant in step S513 .

[0085] Figures 7A - 7C An example of a virtual space image is shown. Here, 3D models are simplified and schematically displayed, but each 3D model is generated in a form corresponding to a corresponding character.

[0086] Figure 7A 700 is a diagram showing an example of a virtual space image. The virtual space image 700 is an example of a schematic camera image of a virtual activity related to a discussion category, and includes 3D models 70 to 72 representing participants. The central 3D model 70 is a 3D model representing a participant assigned the role of a chairman. The 3D models 51 and 52 arranged in each team separated to the left and right are 3D models representing participants assigned the roles of team representative and team member, respectively. The 3D models 70 to 72 present a form corresponding to the importance of the role of each participant.

[0087] Figure 7B701 is a diagram showing another example of a virtual space image. The virtual space image 701 is an example of a schematic camera image of a virtual activity of the demonstration type, and includes 3D models 73 to 75 representing participants. The 3D model 73 on the stage is a 3D model representing a participant assigned the role of a presenter. The 3D model 74 on the sub-stage is a 3D model representing a participant assigned the role of an MC. The 3D model 75 under the stage is a 3D model representing a participant assigned the role of an audience. The 3D models 73 to 75 present a form corresponding to the importance of the role of each participant.

[0088] Figure 7C 702 is a diagram showing another example of a virtual space image. The virtual space image 702 is an example of a schematic camera image of a virtual activity of the entertainment category, and includes 3D models 76 to 78. The 3D model 76 near the front center represents a participant assigned the role of the host. The 3D model 77 near the front side represents a participant assigned the role of the assistant. The 3D model 78 at the back represents a participant assigned the role of the guest. The 3D models 76 to 78 present a form corresponding to the importance of the role of each participant.

[0089] return Figure 5B In step S514, the control unit 113 displays the virtual space image and outputs the sound through the output unit 116. That is, the control unit 113 outputs information for displaying an image of an activity in which the 3D model is arranged in the virtual space to the output unit 116, and the output unit 116 displays the virtual space image and outputs the sound.

[0090] The control unit 113 executes steps S511 to S514 each time it receives a packet of coded information from another terminal device 12. Thus, the participant can hear the voice of the other participant while watching the 3D model of the other participant or the animation of the virtual space image including the other participant and the 3D model of the participant. At this time, by visually recognizing the form of the 3D model corresponding to the importance of the role of the participant, it is easy to focus on the participant with a high importance role. Therefore, it can contribute to the convenience of the participant.

[0091] Figure 8A , 8B They are respectively Figure 5A , 5B A flowchart corresponding to a modified example of Figure 8A , 8B In the example, each terminal device 12 transmits the camera image, distance image, and sound information of the participant. Then, each terminal device 12 generates a 3D model in a form corresponding to the importance of the role of the other participant using the camera image and distance image transmitted from the other terminal devices 12. Figure 8A , 8BIn Figure 5A , 5B the same steps as those of

[0092] are marked with the same reference numerals, and the description thereof is omitted or simplified. Figure 8A In

[0093] , when the control unit 113 determines the form of the 3D model representing the participant in step S502, the control unit 113 transmits 3D model form information representing the form of the 3D model in step S502'. Then, after the control unit 113 performs shooting and sound collection in step S503, in step S505', the control unit 113 encodes the captured image, the distance image, and the sound information to generate encoded information to be transmitted in step S506. Figure 8B In Figure 8B , in step S509, the control unit 113 receives the 3D model form information transmitted from the other terminal device 12 and stores it in the storage unit 112, and acquires the form of the 3D model representing the other participant using the other terminal device 12. In addition, in step S510', the control unit 113 decodes the encoded information transmitted from the other terminal device to acquire the captured image, the distance image, and the sound information. Then, when the control unit 113 generates the 3D model of the other participant in step S511, the control unit 113 generates the 3D model in the form acquired in step S510. While the control unit 113 continues to acquire packets from the other terminal device 12 (No in step S516), the control unit 113 executes steps S510' to S514. If the packet ends (Yes in step S516), the

[0094] steps of Figure 5A , 5B or Figure 8A , 8B are adopted according to the processing performance of the processor of the terminal device 12 and the communication speed, so that load distribution can be achieved among the terminal devices 12.

[0095] In another modification example, the control unit 113 can convert and output the sound pressure and frequency of the voice of a participant with a low importance of the character. For example, the storage unit 112 stores in advance information representing the importance that serves as a reference for voice output. When the importance is lower than the reference importance, for example, when the importance is "low", the control unit 113 performs conversion processing on the voice information of the participant and outputs voice output information for outputting the converted voice. For example, the frequency and sound pressure are adjusted to make it a voice imitating the sound made by a person sucking helium. Then, the output unit 116 receives the voice output information and outputs the converted voice. Thus, it is easy to distinguish the speech voice of a participant with a higher importance character from other speech voices, and therefore the convenience of the participants can be improved.

[0096] In yet another modified example, instead of outputting the voices of participants with a low importance level of the role, the control unit 113 can display a string corresponding to the voice. For example, the storage unit 112 stores in advance information indicating the importance level that serves as a reference for voice output. When the importance level is lower than the reference, for example, when the importance level is "low", the control unit 113 performs voice recognition processing on the voice information of the participant and generates a string corresponding to the voice. Then, the control unit 113 outputs information for displaying the character corresponding to the voice instead of the voice output information. Specifically, when generating the virtual area image, the control unit 113 arranges a string near the corresponding 3D model. Then, the control unit 113 outputs the virtual space image including the string to the output unit 116. Then, the output unit 116 outputs the virtual space image. For example, as Figure 9 shown, in the virtual space image 700 of the virtual event under discussion, a string 90 representing the speech content is arranged and displayed near the 3D model 72 of the participant representing the role of the team member, that is, the participant corresponding to the importance level of "low". It should be noted that the control unit 113 can either generate a string from the voice of the participant according to the importance level of the participant's role, encode the information including the string together with the 3D model and send it to the other terminal device 12, or generate a string from the voice of other participants encoded and sent from the other terminal device 12 according to the importance level of the other participants' roles, and output the information for outputting the generated string. Thus, it is easy to focus on the speech content of the participants with a higher importance level of the role, and therefore the convenience of the participants can be improved.

[0097] In the above, the form of the 3D model is determined based on the importance level of the participant's role. However, the case where the form of the 3D model is arbitrarily associated with each role is also included in the present embodiment. In addition, the number of polygons of the 3D model and the thickness in the optical axis direction are not limited to the three stages as in the above example, and can also be two stages, or four or more stages. In addition, different roles can also correspond to the same number of polygons or thickness.

[0098] In the above, the embodiments have been described based on the respective drawings and examples. However, it should be noted that those skilled in the art can easily make various deformations and corrections based on the present disclosure. Therefore, it is hoped that it is noted that these deformations and corrections are included in the scope of the present disclosure. For example, the functions included in each component, each step, etc. can be reconfigured in a logically consistent manner, and multiple components, steps, etc. can be combined into one, or divided.

Claims

1. A terminal device, comprising: a communication unit; and a control unit that communicates through the communication unit, wherein the control unit receives, from another terminal device, information for generating a 3D model representing a participant who uses the other terminal device to participate in an activity in a virtual space, the information including at least a captured image and a distance image of the participant of the other terminal device, generates the 3D model in a form corresponding to the importance of the role of the participant in the activity, outputs information for displaying an image obtained by rendering a virtual space image captured from a virtual viewpoint, which is the viewpoint of the participant, of the 3D model arranged in the virtual space, the virtual space being a virtual space corresponding to the category of the activity, arranges the 3D model at coordinates in the virtual space corresponding to the role of the participant, and the generated 3D model is represented by a thickness in the optical axis direction corresponding to the distance image; the control unit receives information of the voice of the participant from the other terminal device, and when the importance of the role of the participant is lower than a reference importance, instead of outputting the voice of the participant with a low importance of the role, outputs information for displaying a character corresponding to the voice.

2. The terminal device according to claim 1, wherein the form is the number of polygons for generating the 3D model or the thickness of the 3D model in a predetermined direction.

3. The terminal device according to claim 2, wherein when the role is a first role, the control unit generates the 3D model in a first form, and when the role is a second role with an importance higher than that of the first role, generates the 3D model in a second form in which the number of polygons or the thickness is larger than the first form.

4. The terminal device according to any one of claims 1 to 3, wherein the control unit outputs voice output information for outputting the voice that varies according to the role.

5. An operation method of a terminal device, the terminal device having a communication unit and a control unit that communicates through the communication unit, wherein the operation method includes the following steps: the control unit receives, from another terminal device, information for generating a 3D model representing a participant who uses the other terminal device to participate in an activity in a virtual space, the information including at least a captured image and a distance image of the participant of the other terminal device, generates the 3D model in a form corresponding to the importance of the role of the participant in the activity, outputs information for displaying an image obtained by rendering a virtual space image captured from a virtual viewpoint, which is the viewpoint of the participant, of the 3D model arranged in the virtual space, the virtual space being a virtual space corresponding to the category of the activity, arranges the 3D model at coordinates in the virtual space corresponding to the role of the participant, and the generated 3D model is represented by a thickness in the optical axis direction corresponding to the distance image. The control unit receives information on the voice of the participant from the other terminal device, and when the importance level of the role of the participant is lower than a reference importance level, instead of outputting the voice of the participant with a low importance level of the role, it outputs information for displaying characters corresponding to the voice.

6. The operation method according to claim 5, wherein, the form is the number of polygons for generating the 3D model or the thickness of the 3D model in a predetermined direction.

7. The operation method according to claim 6, wherein, when the role is the first role, the control unit generates the 3D model in the first form, and when the role is the second role with an importance level higher than that of the first role, the control unit generates the 3D model in the second form in which the number of polygons or the thickness is larger than that of the first form.

8. The operation method according to any one of claims 5 to 7, wherein, the control unit outputs voice output information for outputting the voice that varies according to the role.

9. A system having a plurality of terminal devices communicating via a server device, wherein, the first terminal device sends information for generating a 3D model representing a participant who uses the first terminal device to participate in an activity in a virtual space, the information including at least a captured image and a distance image of the participant of the first terminal device, to the second terminal device, the second terminal device generates the 3D model in a form corresponding to the importance level of the role of the participant in the activity, outputs information for displaying an image obtained by rendering a virtual space image captured from a virtual viewpoint as the viewpoint of the participant of the 3D model arranged in the virtual space, the virtual space being a virtual space corresponding to the category of the activity, the 3D model is arranged at coordinates in the virtual space corresponding to the role of the participant, and the generated 3D model is represented by the thickness in the optical axis direction corresponding to the distance image, the second terminal device receives information on the voice of the participant from the first terminal device, and when the importance level of the role of the participant is lower than a reference importance level, instead of outputting the voice of the participant with a low importance level of the role, it outputs information for displaying characters corresponding to the voice.

10. The system according to claim 9, wherein, the form is the number of polygons for generating the 3D model or the thickness of the 3D model in a predetermined direction.

11. The system according to claim 10, wherein, when the role is the first role, the second terminal device generates the 3D model in the first form, and when the role is the second role with an importance level higher than that of the first role, the second terminal device generates the 3D model in the second form in which the number of polygons or the thickness is larger than that of the first form.

12. The system according to any one of claims 9 to 11, wherein, the second terminal device outputs voice output information for outputting the voice that varies according to the role.

Images