Transmission system, transmission method, and transmission program

Abstract

To achieve realistic communication in a virtual space.SOLUTION: A transmission system includes: a plurality of terminal devices that transmits main information including at least one of video and audio in a users' real space and sub information including at least one of video and audio in the users' virtual space, associating them with time; and a server device that acquires the main and sub information transmitted from the respective terminal devices, sets avatar information about the video and audio of avatars of the respective users in the virtual space on the basis of the sub information, and transmits the information to the plurality of terminal devices, determines whether the avatars are in communication with each other on the basis of a state of placement of the avatars in the virtual space, switches the avatar information of an avatar determined to be in communication to a setting based on the main information and transmits the avatar information to the terminal device corresponding to the avatar determined to be in communication.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a transmission system, a transmission method, and a transmission program. [Background Art]

[0002] A technology is known in which a plurality of users communicate with each other via their respective avatars in a virtual space. For example, Patent Document 1 discloses a technology in which participants in a video conference communicate with each other by sharing a scene where avatars displayed on a television discuss with each other. [Prior Art Documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Unexamined Patent Publication No. 11-289524 [Summary of the Invention] [Problems to be Solved by the Invention]

[0004] In the technology for achieving communication by setting avatars in a virtual space as disclosed in Patent Document 1, there is a demand for realizing communication with a sense of presence.

[0005] The present invention has been made in view of the above, and an object of the present invention is to provide a transmission system, a transmission method, and a transmission program that can realize communication with a sense of presence in a virtual space. [Means for Solving the Problems]

[0006] The transmission system according to the present invention comprises a plurality of terminal devices that transmit primary information, which includes at least one of the user's video and audio in the real world, and secondary information, which includes at least one of the user's video and audio in the virtual world, in a time-associated manner; a server device that acquires the primary information and secondary information transmitted from each of the terminal devices, sets avatar information relating to the video and audio of each of the user's avatars in the virtual world based on the secondary information and transmits it to the plurality of terminal devices; determines whether the avatars are in a communication state based on the arrangement of the avatars in the virtual world; and, for terminal devices corresponding to avatars that have been determined to be in a communication state, switches the avatar information of the avatars that have been determined to be in a communication state to a setting based on the primary information and transmits it.

[0007] The transmission method according to the present invention includes the steps of: transmitting to multiple terminal devices a primary information including at least one of the user's video and audio in the real world and secondary information including at least one of the user's video and audio in the virtual world, associated with time; and in a server device, acquiring the primary information and secondary information transmitted from each of the terminal devices, setting avatar information relating to the video and audio of each of the user's avatars in the virtual world based on the secondary information and transmitting it to the multiple terminal devices; determining whether the avatars are in a communication state based on the arrangement of the avatars in the virtual world; and, for terminal devices corresponding to the avatars determined to be in a communication state, switching the avatar information of the avatars determined to be in a communication state to a setting based on the primary information and transmitting it.

[0008] The transmission program according to the present invention causes a computer to perform the following steps: transmit to multiple terminal devices a primary information including at least one of the user's video and audio in the real world and secondary information including at least one of the user's video and audio in the virtual world, associated with time; and in a server device, acquire the primary information and secondary information transmitted from each of the terminal devices, set avatar information relating to the video and audio of each of the user's avatars in the virtual world based on the secondary information and transmit it to the multiple terminal devices; determine whether the avatars are in a communication state based on the arrangement of the avatars in the virtual world; and, for terminal devices corresponding to avatars determined to be in a communication state, switch the avatar information of the avatars determined to be in a communication state to a setting based on the primary information and transmit it. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a transmission system, a transmission method, and a transmission program that enable realistic communication in a virtual space. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a schematic diagram showing an example of a transmission system according to this embodiment. [Figure 2] Figure 2 is a functional block diagram showing an example of the transmission system according to this embodiment. [Figure 3] Figure 3 is a functional block diagram showing an example of the hardware configuration of the information processing device according to this embodiment. [Figure 4] Figure 4 shows an example of the arrangement of avatars in a virtual space. [Figure 5] Figure 5 shows an example of the arrangement of avatars in a virtual space. [Figure 6] Figure 6 shows an example of the arrangement of avatars in a virtual space. [Figure 7]Figure 7 shows an example of information displayed on the display unit of a terminal device. [Figure 8] Figure 8 shows an example of information displayed on the display unit of a terminal device. [Figure 9] Figure 9 shows another example of information displayed on the display unit of a terminal device. [Figure 10] Figure 10 shows another example of information displayed on the display unit of a terminal device. [Figure 11] Figure 11 is a flowchart showing an example of the processing flow in the transmission system according to this embodiment. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the transmission system, transmission method, and transmission program according to the present invention will be described with reference to the drawings. However, the present invention is not limited by these embodiments. Furthermore, the components in the following embodiments include those that are easily substituted or substantially identical to those that are easily substituted by those skilled in the art.

[0012] Figure 1 is a schematic diagram showing an example of the transmission system 100 according to this embodiment. Figure 2 is a functional block diagram showing an example of the transmission system 100 according to this embodiment. As shown in Figures 1 and 2, the transmission system 100 according to this embodiment comprises a plurality of terminal devices 10 and a server device 20. In the transmission system 100 shown in Figures 1 and 2, the system utilizes a virtual space provided by the server device 20 by accessing the server device 20 via a network from the terminal devices 10. Examples of such virtual spaces include various virtual spaces corresponding to real spaces, such as offices, conference rooms for web conferences, stores, and shopping malls.

[0013] Examples of multiple terminal devices 10 include information terminals such as notebook personal computers, desktop personal computers, tablets, and smartphones. Each terminal device 10 has a shooting unit 11, an audio input unit 12, an operation unit 13, a display unit 14, an audio output unit 15, and a control unit 16.

[0014] The imaging unit 11 captures an image of a user of the terminal device 10 and generates imaging information. The imaging unit 11 outputs the generated imaging information to the control unit 16. The imaging unit 11 is constituted by an imaging device such as a visible light camera, a far-infrared camera, or a near-infrared camera, for example. The imaging unit 11 may be constituted by a combination of a visible light camera, a far-infrared camera, and a near-infrared camera, for example.

[0015] The voice input unit 12 collects sounds such as the voice of the user of the terminal device 10 and sounds around the user, and generates collected sound information. The voice input unit 12 transmits the generated collected sound information to the control unit 16. The voice input unit 12 is constituted by a sound collection device such as a microphone, for example.

[0016] The operation unit 13 receives various operations from a user on the terminal device 10, and outputs the operations as an operation signal to the control unit 16. As the operation unit 11, an input device such as a mouse, a keyboard, a touch panel, a button, a lever, a dial, or a switch is used, for example.

[0017] The display unit 14 displays various types of information. Examples of the display unit 14 include a liquid crystal display, an organic EL (Electro-Luminescence) display, and the like. The display unit 14 may be, for example, a head-mounted display worn on the user's head. The display unit 14 displays video based on a video signal output from the control unit 16.

[0018] The audio output unit 15 is a device that outputs various types of collected sound information. The audio output unit 15 may be a speaker externally connected to the control unit 16, or may be a speaker built into a housing that accommodates the control unit 16. The audio output unit 15 outputs audio based on an audio signal output from the control unit 16.

[0019] The control unit 16 comprehensively controls the operation of the terminal device 10. The control unit 16 includes a communication unit 17, a processing unit 18, and a storage unit 19.

[0020] The communication unit 17 communicates with external devices via wired or wireless means. The communication unit 17 also communicates with the server device 20. The communication unit 17 transmits user primary and secondary information to the server device 20 under the control of the processing unit 18, which will be described later. The communication unit 17 outputs spatial display information, audio output information, and avatar information of the virtual space received from the server device 20 to the processing unit 18. The spatial display information is information for displaying the image of the virtual space on the display unit 14. The audio output information is information for outputting audio from the virtual space through the audio output unit 15. The avatar information includes information for displaying the avatar in the virtual space on the display unit 14 and information for outputting audio from the avatar through the audio output unit 15.

[0021] The processing unit 18 performs various processes. The processing unit 18 acquires spatial display information, audio output information, and avatar information received by the communication unit 17. Based on the acquired spatial display information, audio output information, and avatar information, the processing unit 18 displays the video of the virtual space and the video of the avatar on the display unit 14, and outputs the audio of the virtual space and the audio emitted from the avatar from the audio output unit 15.

[0022] The processing unit 18 generates primary and secondary user information based on the shooting information from the shooting unit 11, the sound acquisition information from the audio input unit 12, and the operation signals from the operation unit 13. Here, the primary information includes at least one of the user's video and audio in the real world. The secondary information includes at least one of the user's video and audio in the virtual world. The primary and secondary information is used when the server device 20 sets the avatar information. The processing unit 18 controls the communication unit 17 to transmit the generated primary and secondary information to the server device 20 in association with the time.

[0023] For example, when a user controls the movement of an avatar in a virtual space using the operation unit 13, the processing unit 18 generates operation information for the avatar according to the operation content. The processing unit 18 then transmits the generated operation information from the communication unit 17 to the server device 20.

[0024] The storage unit 19 stores various types of information. The storage unit 19 stores programs, data, etc., for processing various types of data in the processing unit 18. The storage unit 19 stores a program that causes a computer to perform the step of transmitting, in a time-associated manner, primary information, which includes at least one of the user's real-world video and audio, and secondary information, which includes at least one of the user's virtual-world video and audio.

[0025] The server device 20 includes a communication unit 21, a processing unit 22, and a storage unit 23.

[0026] The communication unit 21 communicates information with each terminal device 10 via wired or wireless means.

[0027] The processing unit 22 performs various processes, including processes for operating and managing the virtual space. When the terminal device 10 accesses the virtual space, the processing unit 22 causes the communication unit 21 to transmit virtual space display information and audio output information to the terminal device 10. The processing unit 22 also acquires the main information and sub-information transmitted from each terminal device 10 and received by the communication unit 21.

[0028] The processing unit 22 sets the user's avatar information in the virtual space based on at least one of the acquired primary information and secondary information, and causes the communication unit 21 to transmit the set avatar information to the terminal device 10.

[0029] In this case, the processing unit 22 initially sets the avatar information based on the sub-information and has the communication unit 21 transmit the set avatar information to the terminal device 10. After setting the initial avatar information, the processing unit 22 determines whether the avatars in the virtual space are in a communication state, and for terminal devices 10 corresponding to avatars in a communication state, it switches the avatar information to a setting based on the main information and transmits it.

[0030] The processing unit 22 determines whether avatars are in a state of communication based on their arrangement in the virtual space. The arrangement includes, for example, the avatars' positions in the virtual space, the direction they are facing, their posture, and whether they are moving. For instance, the processing unit 22 can determine that avatars are in a state of communication if their distance in the virtual space is less than or equal to a predetermined value. Furthermore, the processing unit 22 can determine that avatars are in a state of communication if they are facing each other in the virtual space. Additionally, if one avatar speaks to another in the virtual space, the processing unit 22 can determine that both avatars are in a state of communication.

[0031] The processing unit 22, for terminal devices 10 corresponding to avatars it determines are in a communication state, switches the video and audio of the avatars determined to be in a communication state to settings based on the primary information and transmits them. In other words, between users attempting to communicate in the virtual space, the avatar's video is set to settings based on the primary information. On the other hand, the processing unit 22 transmits the video and audio of these avatars determined to be in a communication state to terminal devices 10 corresponding to avatars it determines are not in a communication state, using settings based on the secondary information. In other words, for other users who do not communicate with these users in the virtual space, the avatar's video and audio settings do not change. By displaying avatars in a communication state separately from other avatars in this way, a more immersive display can be achieved.

[0032] The processing unit 22 can determine the degree of communication between avatars that it has determined are in a communication state, based on their arrangement. In this case, the processing unit 22 can set the degree to which the main information is reflected according to the degree of communication. For example, the processing unit 22 can increase the degree to which the main information is reflected as the distance between avatars decreases. In this case, the processing unit 22 can gradually increase the degree to which the main information is reflected, for example, by setting only one of the voice and avatar outlines based on the main information, setting both the voice and avatar outlines based on the main information, or setting all of the voice and avatar information based on the main information.

[0033] The memory unit 23 stores various information. The memory unit 23 stores various programs, data, etc., for processing various operations in the processing unit 22. For example, the memory unit 23 stores background information in the virtual space. The memory unit 23 stores a program that causes the computer to perform the following steps in the server device 20: acquire primary and secondary information transmitted from each terminal device 10, set avatar information regarding the video and audio of each user's avatar in the virtual space based on the secondary information and transmit it to multiple terminal devices 10, determine whether the avatars are in a communication state based on the arrangement of the avatars in the virtual space, and for terminal devices 10 corresponding to avatars determined to be in a communication state, switch the avatar information of the avatars determined to be in a communication state to a setting based on the primary information and transmit it.

[0034] Figure 3 is a functional block diagram showing an example of the hardware configuration of the information processing device according to this embodiment. The terminal device 10 (control unit 16) and the server device 20 each include the information processing device 1. The information processing device 1 has a processor 2, a memory 3, a storage 4, and an interface 5. The processor 2, memory 3, storage 4, and interface 5 are interconnected by a bus or the like.

[0035] Processor 2 includes, for example, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Memory 3 includes, for example, non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory). Storage 4 includes, for example, a storage device such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive). Storage 4 stores programs for realizing the functions of the terminal device 10 (control unit 16) and the server device 20 described above. Interface 5 includes input / output circuits such as a network interface card. Interface 5 communicates with external devices.

[0036] Processor 2 reads each program stored in storage 4 and loads it into memory 3, thereby executing processing according to each of the above functions. Information processing device 1 operates as a computer that performs various information processing by reading and executing programs in this manner.

[0037] Furthermore, the above program is not limited to being stored in storage 4. For example, the program may be distributed to the information processing device 1 via a network. Alternatively, the program may be recorded on an external recording medium and read out and distributed to the information processing device 1. Also, the program is not limited to being executed by the information processing device 1. For example, another information processing device different from the information processing device 1 may execute the program, or the information processing device 1 and another information processing device may cooperate to execute the program.

[0038] Next, the operation of the transmission system 100 configured as described above will be explained. When a user who wishes to use the virtual space provided by the server device 20 accesses the server device 20 via the terminal device 10, the processing unit 22 of the server device 20 causes the communication unit 21 to transmit spatial display information and audio output information about the virtual space to the terminal device 10.

[0039] In the terminal device 10, the communication unit 17 receives spatial display information and audio output information transmitted from the server device 20. The processing unit 18 acquires the received spatial display information and audio output information, displays the background of the virtual space on the display unit 14 based on the acquired spatial display information, and outputs audio from the virtual space from the audio output unit 15 based on the audio output information.

[0040] In the terminal device 10, the camera unit 11 captures the user's appearance, the voice input unit 12 captures the user's voice and other sounds, and the operation unit 13 accepts user input. The camera unit 11 outputs the captured information to the control unit 16. The voice input unit 12 outputs the captured sound information to the control unit 16. The operation unit 13 outputs operation signals to the control unit 16.

[0041] The processing unit 18 acquires shooting information, sound acquisition information, and operation signals, and generates main information and sub-information based on the acquired information and signals. In this state, if, for example, an operation to set a user's avatar in the virtual space is input by the operation unit 13, the processing unit 18 sends the generated main information and sub-information, associated with the time, from the communication unit 17 to the server device 20.

[0042] In the server device 20, the communication unit 21 receives the main information and sub-information transmitted from the terminal device 10. The processing unit 22 acquires the main information and sub-information transmitted from the terminal device 10. In the initial state, the processing unit 22 sets the avatar information based on the sub-information. That is, it sets the avatar information using only the sub-information of the main information and sub-information associated with the time. The processing unit 22 transmits the set avatar information to the terminal device 10. In the initial state, if the main information and sub-information are acquired from multiple terminal devices 10, the processing unit 22 sets all the avatar information based on the sub-information and transmits all the set avatar information to each terminal device 10.

[0043] In each terminal device 10, the communication unit 17 receives avatar information transmitted from the server device 20. Based on the received avatar information, the processing unit 18 displays the avatar's image on the display unit 14 and outputs the avatar's voice from the audio output unit 15. In the initial state, the avatar's image based on the sub-information is displayed on the display unit 14, and the avatar's voice based on the sub-information is output from the audio output unit 15. If multiple users' avatars exist in the virtual space, in the initial state, all avatars are displayed on the display unit 14 in a manner based on the sub-information, and voice is output from the audio output unit 15.

[0044] The user may operate the control unit 13 to move their avatar in the virtual space and attempt to communicate with other avatars, for example, by talking to them. In the server device 20, the processing unit 22 determines whether or not the avatars in the virtual space are in a communication state.

[0045] In this case, the processing unit 22 determines whether the avatars are in a communication state based on their arrangement in the virtual space. Examples of the arrangement state include the avatars' positions in the virtual space, the direction the avatars are facing in the virtual space, and the avatars' postures in the virtual space.

[0046] Figures 4 to 6 show examples of avatar placement in a virtual space. The background image of the virtual space is omitted from the illustrations in Figures 4 to 6. As shown in Figure 4, the processing unit 22 can determine whether avatars are in a state of communication based on the distance between them in the virtual space. For example, the processing unit 22 can determine that avatars are in a state of communication if the distance between them in the virtual space is less than or equal to a predetermined value. In the example shown in Figure 4, the processing unit 22 can determine that avatar A1 and avatar A2 are in a state of communication. Furthermore, it can determine that avatars A3 and A4 are not in a state of communication with other avatars.

[0047] Furthermore, as shown in Figure 5, the processing unit 22 can determine whether avatars are in a state of communication with each other based on the orientation of the avatars in the virtual space. For example, if avatars are facing each other in the virtual space, the processing unit 22 can determine that these avatars are in a state of communication. In the example shown in Figure 5, the processing unit 22 can determine that avatar A5 and avatar A6 are in a state of communication. It can also determine that avatars A7 and A8 are not in a state of communication with the other avatars.

[0048] Furthermore, as shown in Figure 6, the processing unit 22 can determine whether avatars are in a state of communication with each other based on the actions of the avatars in the virtual space. For example, if one avatar speaks to another avatar in the virtual space, the processing unit 22 can determine that both the first avatar and the other avatar are in a state of communication. In the example shown in Figure 6, the processing unit 22 can determine that avatar A9 and avatar A10 are in a state of communication. It can also determine that avatars A11 and A12 are not in a state of communication with the other avatar. In addition to the above, the processing unit 22 can determine that both the first avatar and the other avatar are in a state of communication if, for example, one avatar points at another avatar, touches or attempts to touch another avatar, or waves at another avatar, or if a user operating one avatar selects another avatar using the operation unit 13 described above.

[0049] The processing unit 22 may determine that a communication state exists if two or more of the cases shown in Figures 4 to 6 (distance between avatars in the virtual space, orientation, and actions) are met. In this case, the processing unit 22 may set a priority order for the conditions for determining a communication state, for example, by first making a determination based on the distance between avatars, then determining whether the avatars are facing each other if the distance is less than a predetermined value, and finally determining that a communication state exists if they are facing each other.

[0050] The processing unit 22 transmits the avatar information of the avatars determined to be in a communication state to the terminal device 10 corresponding to the avatars, switching the settings to those based on the primary information. On the other hand, the processing unit 22 transmits the avatar information of the avatars determined to be in a communication state to the terminal device 10 corresponding to the avatars determined to be in a communication state, with the settings based on the secondary information.

[0051] Figures 7 and 8 show examples of information displayed on the display unit 14 of the terminal device 10. While Figures 7 and 8 show the case corresponding to Figure 4 among the examples shown in Figures 4 to 6 above, a similar explanation is possible for the cases corresponding to Figures 5 and 6. Figure 7 shows an example of the display unit 14 in the terminal device 10 of a user attempting to communicate in a virtual space. As shown in Figure 7, in the terminal device 10 corresponding to avatars determined to be in a communication state (avatars A1 and A2 in the example of Figure 4), the avatar's image is displayed and sound is output based on the avatar information set based on the main information. That is, between users attempting to communicate in a virtual space, the image and sound of avatars A1 and A2 are switched to settings based on the main information.

[0052] Figure 8 shows an example of the display unit 14 in the terminal device 10 for a user who does not communicate in the virtual space. As shown in Figure 8, in the terminal device 10 corresponding to avatars that are determined not to be in a communication state (avatars A3 and A4 in the example of Figure 4), the avatar's image is displayed and sound is output based on the avatar information set based on the sub-information. In other words, for users who do not communicate with other users in the virtual space, the settings for the avatar's image and sound do not change. By displaying avatars that are in a communication state separately from other avatars in this way, a more immersive display can be achieved.

[0053] The processing unit 22 can determine the degree of communication between avatars that it has determined are in a communication state, based on their arrangement. In this case, the processing unit 22 can set the degree to which the main information is reflected according to the degree of communication.

[0054] Figures 9 and 10 show other examples of information displayed on the display unit 14 of the terminal device 10. Figures 9 and 10 omit the illustration of the background image of the virtual space. Although Figures 9 and 10 show the case corresponding to Figure 4 among the examples shown in Figures 4 to 6 above, a similar explanation can be given for the cases corresponding to Figures 5 and 6. The processing unit 22 can increase the degree to which it reflects the main information as the distance between avatars decreases. In this case, the processing unit 22 may, when the distance between avatars becomes closer than a first threshold, set only the inner face portion of the avatar's appearance based on the main information as shown in Figure 9, and when the distance between avatars becomes closer than a second threshold which is smaller than the first threshold, set the entire appearance of the avatar based on the main information as shown in Figure 10, and so on, gradually increase the degree to which it reflects the main information in the display of the avatar's appearance. Furthermore, from the state shown in Figure 10, the degree to which it reflects the main information in both the appearance display and the sound may be gradually increased, for example, by setting the avatar's voice based on the main information.

[0055] Figure 11 is a flowchart showing an example of the processing flow in the transmission system 100 according to this embodiment.

[0056] As shown in Figure 11, when an access request to the server device 20 is made from the terminal device 10 to use the virtual space, the processing unit 22 causes the communication unit 21 to transmit spatial display information and audio output information about the virtual space to the terminal device 10 (step S101). The terminal device 10 receives the spatial display information and audio output information transmitted from the server device 20 via the communication unit 17. The processing unit 18 acquires the received spatial display information and audio output information (step S102). Based on the acquired spatial display information, the processing unit 18 displays the background of the virtual space on the display unit 14 and outputs audio from the virtual space from the audio output unit 15 based on the audio output information (step S103).

[0057] On the terminal device 10 side, the camera unit 11 captures the user's appearance, the voice input unit 12 captures the user's voice and other sounds, and the operation unit 13 accepts user operations (step S104). The camera unit 11 outputs the captured information to the control unit 16. The voice input unit 12 outputs the captured sound information to the control unit 16. The operation unit 13 outputs operation signals to the control unit 16.

[0058] In the control unit 16, the processing unit 18 acquires shooting information, sound acquisition information, and operation signals, and generates main information and sub-information based on the acquired information and signals (step S105). In response to input from the operation unit 13, the processing unit 18 causes the communication unit 17 to transmit the generated main information and sub-information to the server device 20 in association with the time (step S106).

[0059] In the server device 20, the main information and sub-information transmitted from the terminal device 10 are received by the communication unit 21. The processing unit 22 acquires the main information and sub-information received by the receiving unit 21 (step S107). The processing unit 22 sets the avatar information for the avatar corresponding to each terminal device 10 in the initial state based on the sub-information, and transmits the set avatar information to the terminal device 10 (step S108).

[0060] In each terminal device 10, the avatar information transmitted from the server device 20 is received by the communication unit 17 (step S109). Based on the received avatar information, the processing unit 18 displays the avatar's image on the display unit 14 and outputs the avatar's voice from the audio output unit 15 (step S110).

[0061] Subsequently, in the server device 20, the processing unit 22 determines whether or not the avatars are in a communication state in the virtual space (step S111). If the processing unit 22 determines that they are in a communication state (Yes in step S111), it switches the avatar information of the avatars that it determined to be in a communication state to the terminal device 10 corresponding to the avatars and sends it to the terminal device 10 (step S112). If the processing unit 22 determines that they are not in a communication state (No in step S111), it skips the process in step S112, that is, it maintains the avatar information settings.

[0062] When avatar information after a setting change is transmitted from the server device 20 to each terminal device 10, the communication unit 17 receives the avatar information. If the processing unit 18 obtains the avatar information after a setting change (Yes in step S113), it displays the avatar's image on the display unit 14 and outputs the avatar's voice from the audio output unit 15 based on the obtained avatar information after a setting change (step S114). If the processing unit 18 does not obtain the avatar information after a setting change (No in step S113), it skips the process in step S114, that is, it maintains the display of the avatar's image and the output of its voice.

[0063] As described above, the transmission system 100 according to this embodiment includes a plurality of terminal devices 10 that transmit primary information including at least one of the user's video and audio in the real world, and secondary information including at least one of the user's video and audio in the virtual world, associated with time; and a server device 20 that acquires the primary and secondary information transmitted from each terminal device 10, sets avatar information relating to the video and audio of each user's avatar in the virtual world based on the secondary information and transmits it to the plurality of terminal devices 10, determines whether the avatars are in a communication state based on the arrangement of the avatars in the virtual world, and for terminal devices 10 corresponding to avatars that have been determined to be in a communication state, switches the avatar information of the avatars determined to be in a communication state to a setting based on the primary information and transmits it.

[0064] Furthermore, the transmission method according to this embodiment includes the steps of: transmitting primary information, which includes at least one of the user's video and audio in the real world, and secondary information, which includes at least one of the user's video and audio in the virtual world, to multiple terminal devices 10, associating them with time; and transmitting the primary and secondary information transmitted from each terminal device 10 to the multiple terminal devices 10 in the server device 20, setting avatar information relating to the video and audio of each user's avatar in the virtual world based on the secondary information, and transmitting it to the multiple terminal devices 10, determining whether the avatars are in a communication state based on the arrangement of the avatars in the virtual world, and for the terminal devices 10 corresponding to the avatars that have been determined to be in a communication state, switching the avatar information of the avatars determined to be in a communication state to a setting based on the primary information and transmitting it.

[0065] Furthermore, the transmission program according to this embodiment causes a computer to perform the following steps: a step of transmitting to multiple terminal devices 10, in association with time, primary information including at least one of the user's video and audio in the real world, and secondary information including at least one of the user's video and audio in the virtual world; and a step of the server device 20 acquiring the primary and secondary information transmitted from each terminal device 10, setting avatar information relating to the video and audio of each user's avatar in the virtual world based on the secondary information and transmitting it to the multiple terminal devices 10, determining whether the avatars are in a communication state based on the arrangement of the avatars in the virtual world, and for terminal devices 10 corresponding to avatars determined to be in a communication state, switching the avatar information of the avatars determined to be in a communication state to a setting based on the primary information and transmitting it.

[0066] In this configuration, when users attempt to communicate in the virtual space, the avatar's video and audio settings switch to those based on the primary information. On the other hand, for users who do not communicate with other users in the virtual space, the avatar's video and audio settings do not switch. By distinguishing and configuring avatars in communication mode from other avatars in this way, a more immersive communication experience can be achieved.

[0067] In the transmission system 100 according to this embodiment, the server device 20 determines that avatars are in a communication state if the distance between them in the virtual space is less than or equal to a predetermined value. With this configuration, the communication state between avatars can be appropriately determined.

[0068] In the transmission system 100 according to this embodiment, the server device 20 determines that avatars facing each other in the virtual space are in a communication state. With this configuration, the communication state between avatars can be appropriately determined.

[0069] In the transmission system 100 according to this embodiment, when one avatar speaks to another avatar in the virtual space, the server device 20 determines that both the first avatar and the second avatar are in a communication state. This configuration allows for the appropriate determination of the communication state between avatars.

[0070] In the transmission system 100 according to this embodiment, the server device 20 determines the degree of communication between avatars that it has determined are in a communication state, based on their arrangement, and sets the degree to which main information is reflected according to the degree of communication. With this configuration, since the degree to which main information is reflected differs depending on the degree of communication, it becomes possible to realize more realistic communication.

[0071] The technical scope of the present invention is not limited to the embodiments described above, and modifications can be made as appropriate without departing from the spirit of the invention. [Explanation of Symbols]

[0072] 1...Information processing device, 2...Processor, 3...Memory, 4...Storage, 5...Interface, 10...Terminal device, 11...Shooting unit, 12...Audio input unit, 13...Operation unit, 14...Display unit, 15...Audio output unit, 16...Control unit, 17,21...Communication unit, 18,22...Processing unit, 19,23...Storage unit, 20...Server device, 100...Transmission system

Claims

1. Multiple terminal devices that transmit primary information, which includes at least one of the user's video and audio in the real world, and secondary information, which includes at least one of the user's video and audio in the virtual world, in a time-associated manner. A server device that acquires the primary information and secondary information transmitted from each of the terminal devices, sets avatar information relating to the video and audio of each user's avatar in the virtual space based on the secondary information and transmits it to multiple terminal devices, determines whether the avatars are in a communication state based on the arrangement of the avatars in the virtual space, and transmits to the terminal device corresponding to the avatar that has been determined to be in a communication state, switching the avatar information of the avatar determined to be in a communication state to a setting based on the primary information. Equipped with, The server device determines that the avatars facing each other in the virtual space are in a state of communication. Transmission system.

2. A plurality of terminal devices that transmit primary information including at least one of video and audio in the user's real space and secondary information including at least one of video and audio in the user's virtual space, associated with time, A server device that acquires the primary information and secondary information transmitted from each of the terminal devices, sets avatar information relating to the video and audio of each user's avatar in the virtual space based on the secondary information and transmits it to multiple terminal devices, determines whether the avatars are in a communication state based on the arrangement of the avatars in the virtual space, and transmits to the terminal device corresponding to the avatar that has been determined to be in a communication state, switching the avatar information of the avatar determined to be in a communication state to a setting based on the primary information. Equipped with, When one avatar speaks to another avatar in the virtual space, the server device determines that both the first avatar and the second avatar are in a communication state. Transmission system.

3. A plurality of terminal devices that transmit primary information including at least one of the user's real-world video and audio, and secondary information including at least one of the user's virtual-world video and audio, associated with time, A server device that acquires the primary information and secondary information transmitted from each of the terminal devices, sets avatar information relating to the video and audio of each user's avatar in the virtual space based on the secondary information and transmits it to multiple terminal devices, determines whether the avatars are in a communication state based on the arrangement of the avatars in the virtual space, and transmits to the terminal device corresponding to the avatar that has been determined to be in a communication state, switching the avatar information of the avatar determined to be in a communication state to a setting based on the primary information. Equipped with, The server device determines the degree of communication between the avatars that it has determined to be in a communication state, based on their arrangement, and sets the degree to which the main information is reflected according to the degree of communication. Transmission system.

4. The server device determines that communication is in progress for avatars whose distance in the virtual space is less than or equal to a predetermined value. The transmission system according to any one of claims 1 to 3.

5. The steps include transmitting, in multiple terminal devices, primary information including at least one of the user's video and audio in the real world, and secondary information including at least one of the user's video and audio in the virtual world, associated with time; The server device acquires the primary information and secondary information transmitted from each of the terminal devices, sets avatar information relating to the video and audio of each user's avatar in the virtual space based on the secondary information and transmits it to the multiple terminal devices, determines whether the avatars are in a communication state based on the arrangement of the avatars in the virtual space, and for the terminal devices corresponding to the avatars determined to be in a communication state, switches the avatar information of the avatars determined to be in a communication state to the setting based on the primary information and transmits it. Includes, The server device determines the degree of communication between the avatars that it has determined to be in a communication state, based on their arrangement, and sets the degree to which the main information is reflected according to the degree of communication. Transmission method.

6. The steps include transmitting, in multiple terminal devices, primary information including at least one of the user's video and audio in the real world, and secondary information including at least one of the user's video and audio in the virtual world, associated with time; The server device acquires the primary information and secondary information transmitted from each of the terminal devices, sets avatar information relating to the video and audio of each user's avatar in the virtual space based on the secondary information and transmits it to the multiple terminal devices, determines whether the avatars are in a communication state based on the arrangement of the avatars in the virtual space, and for the terminal devices corresponding to the avatars determined to be in a communication state, switches the avatar information of the avatars determined to be in a communication state to the setting based on the primary information and transmits it. Have the computer run it, The server device determines the degree of communication between the avatars that it has determined to be in a communication state, based on their arrangement, and sets the degree to which the main information is reflected according to the degree of communication. Transmission program.

Images