Information processing system, edge device, server, control method, program, and storage medium

Abstract

To provide an information processing system, an edge device, a server, a control method, a program, and a storage medium that enable a user to establish a conversation with a desired person through an avatar even in a noisy surrounding situation in a virtual space.SOLUTION: An information processing system 1 is configured to determine that first voice data having a first avatar associated with a first user terminal as a sound source is associated with a second avatar associated with a second user terminal, and analyze the first voice data and the second voice data excluding the first voice data among the voice data transmitted to the second user terminal, and if it is analyzed that the second voice data interferes with the first voice data, adjust the volume of the voice data transmitted to the second user terminal.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing system, an edge device, a server, a control method, a program, and a storage medium, and more particularly to an information processing system, an edge device, a server, a control method, a program, and a storage medium for controlling communication of avatars in a virtual space. [Background technology]

[0002] It is becoming common for multiple users to wear HMDs (Head Mounted Displays) and communicate with each other through their avatars in a virtual space created by VR (Virtual Reality). Voice chat is often used as a means of communication between users wearing HMDs.

[0003] Patent Document 1 discloses a technique for imparting directionality to the sound output from a sound source object of a first avatar and setting the volume parameters of the output sound according to the position of a sound collection object of a second avatar. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6289703 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, consider a situation where a user and a friend participate in a live music event held in a virtual space as audience members, each wearing an HMD. Live music events are usually noisy around the users, with the voices of the performers singing and playing, and the cheers of other audience members. Therefore, even if the friend talks to the user while participating in the live music event in the virtual space, it is difficult for the user to hear the friend's voice, making it difficult to hold a conversation.

[0006] If the technology of Patent Document 1 is applied in this situation, several problems are expected. The first problem is that since the volume parameter setting can only be controlled on the sound source object side, even if the user of the second avatar (sound collection object side) thinks that "I can't hear the sound output from the first avatar (sound source object side)," it cannot be solved. The second problem is that the user of the first avatar (sound source object side) must decide whether or not it is necessary to change the volume parameter setting. The third problem is that if avatars of other spectators are present near the target acquaintance (second avatar), the loud voice output from the sound source object of the first avatar will reach the other spectators, causing a nuisance.

[0007] Therefore, the present invention aims to provide an information processing system, edge device, server, control method, program, and storage medium that enable users to hold a conversation with a desired person using an avatar, even in noisy situations in a virtual space. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, an information processing system according to claim 1 of the present invention is an information processing system comprising three or more user terminals connected via a network, each having an audio input unit and an audio output unit, and providing a virtual space including avatars linked to each of the plurality of user terminals, the information processing system including: audio generation means for generating audio data using each avatar linked to each of the plurality of user terminals as a sound source based on audio input to the audio input unit of each of the plurality of user terminals; audio data transmission means for transmitting the audio data using each avatar as a sound source to the plurality of user terminals; and a first audio data using a first avatar linked to a first user terminal among the plurality of user terminals as a sound source, the first audio data being transmitted to the plurality of user terminals via audio data transmission means. the voice data analysis means for analyzing the first voice data and second voice data excluding the first voice data from the voice data transmitted to the second user terminal by the voice data transmitting means; and a volume adjustment means for adjusting the volume of the voice data transmitted to the second user terminal by the voice data transmission means, the volume being output by the voice output unit of the second user terminal when the first voice data is determined to be associated with the second avatar by the association determination means and the voice data analysis means analyzes that the second voice data interferes with the first voice data.

[0009] In order to solve the above problem, an edge device according to claim 18 of the present invention is an edge device that functions as one of the multiple user terminals equipped in the information processing system, and is characterized in having the volume adjustment means.

[0010] In order to solve the above problem, the server of claim 19 of the present invention is a server provided in the information processing system, connected to the multiple user terminals via the network, and characterized in that it comprises the voice data transmission means, the relatedness determination means, and the voice data analysis means. Effect of the Invention

[0011] According to the present invention, even in a noisy virtual space, users can have a conversation with a desired person using an avatar. [Brief description of the drawings]

[0012] [Figure 1A] FIG. 2 is a block diagram showing an example of a hardware configuration of a user terminal as an edge device according to the present embodiment. [Figure 1B] FIG. 2 is a block diagram showing an example of a hardware configuration of a server according to the embodiment. [Figure 1C] 1 is a block diagram showing an example of an overall functional configuration of an information processing system according to an embodiment of the present invention, which has a user terminal and a server. [Diagram 2] 11 is a flowchart showing an example of a process for adjusting the volume of audio data according to a situation in a virtual space executed in the information processing system. [Figure 3A] 3 is a flowchart showing an example of the sound adjustment process in step S204 of FIG. 2. [Figure 3B] 3B is a flowchart showing an example of the relatedness determination process in step S301 of FIG. 3A. [Figure 3C] 3B is a flowchart showing an example of the voice data analysis process in step S30 of FIG. 3A. [Figure 4] This is a diagram showing an example of a scene in a VR virtual space provided by the server. [Diagram 5] 13 is a table showing an example of a friend registration status list between avatars, which is registered in advance as data by a server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] 1A to 1C are block diagrams showing an example of the configuration of a server 111 according to this embodiment and a plurality of user terminals (hereinafter collectively referred to as "user terminals 100") as edge devices according to this embodiment connected thereto.

[0015] 1A is a block diagram showing an example of a hardware configuration of a user terminal 100. The user terminal 100 according to this embodiment is an information processing device including a CPU 101a, a ROM 102a, a RAM 103a, and a HDD 104a, and each functional unit is communicatively connected via a bus 105a.

[0016] The CPU 101a is a central processing unit that executes various processes using programs and data stored in the RAM 103a or the ROM 102a. As a result, the CPU 101a controls the operation of the entire user terminal 100, and executes or controls the various processes that have been described as being executed by the user terminal 100.

[0017] The ROM 102a is a read-only memory, and stores setting data for the user terminal 100, computer programs or data related to the startup of the user terminal 100, computer programs or data related to the basic operation of the user terminal 100, and the like.

[0018] The RAM 103a is a random access memory and has an area for storing computer programs or data loaded from the ROM 102a or the HDD (Hard Disk Drive) 104a. The RAM 103a also has a work area used when the CPU 101a executes various processes. In this way, the RAM 103a can provide various areas as needed.

[0019] The HDD 104a is an example of a large-capacity information storage device. The HDD 104a stores an OS (Operating System), or computer programs or data for causing the CPU 101a to execute and control various processes described as being performed by the user terminal 100. In the above description, the registration and storage of various information may be performed by the HDD 104a or the RAM 103a. The computer programs or data stored in the HDD 104a are loaded into the RAM 103a as appropriate under the control of the CPU 101a, and are processed by the CPU 101a.

[0020] In addition to or instead of the HDD 104a, a medium (recording medium) and a drive device for reading and writing computer programs and data from and to the medium may be provided. Known examples of such media include flexible disks (FDs), CD-ROMs, DVDs, USB memories, MOs, and flash memories.

[0021] The hardware configuration of an information processing device applicable to the user terminal 100 is not limited to the configuration shown in FIG. 1A, and can be modified / changed as appropriate. The user terminal 100 is further connected to external devices including an HMD 106, a microphone 107, a speaker 108, a controller 109, and a network 110 so as to be able to communicate with them. However, the user terminal 100 may have some or all of the functions of these external devices. For example, while the HMD 106 and the user terminal 100 are separate devices in FIG. 1A, the HMD 106 and the user terminal 100 may be integrated to configure one user terminal 100.

[0022] The HMD 106 has a function of displaying a visual field image in a virtual space by the VR provided by the server 111. The HMD 106 also has various built-in sensors (first and second detection units) for detecting the inclination of the HMD 106 itself and the line of sight and hand movements of a user (hereinafter, the wearing user) wearing the HMD 106. The CPU 101 (line of sight direction setting means) sets the line of sight direction of the wearing user's avatar based on the position of the wearing user's avatar and the line of sight of the wearing user detected by the HMD 106, and determines the visual field image displayed by the HMD 106 according to the line of sight direction. Furthermore, the CPU 101 (avatar operation control means) moves the hands of the wearing user's avatar displayed in the virtual space according to the hand movements of the wearing user detected by the HMD 106. Note that the server 111 may set the line of sight direction of the wearing user in the virtual space, determine the visual field image, and control the hand movements of the wearing user's avatar based on the detection results of the various sensors of the HMD 106.

[0023] When the microphone 107 (audio input unit / audio generation means) receives input of the voice of the wearing user, it generates the audio data and transmits it to the CPU 101a. The CPU 101a transmits the audio data from the microphone 107 to the server 111 as audio data whose sound source is an avatar linked to the user terminal 100. If the wearing user is a performer in a live music event in the virtual space, the microphone 107 receives input of the singing voice or the sound of the performance. The speaker 108 (audio output unit) outputs the audio data collected in the virtual space.

[0024] The controller 109 (setting unit) accepts input from the user wearing the device and reflects the input in moving the position of an avatar, which represents the user wearing the device in the virtual space, and in operating a UI (User Interface) displayed in the virtual space.

[0025] FIG. 1B is a block diagram showing an example of a hardware configuration of the server 111. As shown in FIG.

[0026] Unlike the user terminal 100, the server 111 is not connected to external devices such as the HMD 106, microphone 107, speaker 108, and controller 109, but is an information processing device with the same hardware configuration as the user terminal 100. That is, the server 111 includes a CPU 101b, a ROM 102b, a RAM 103b, and a HDD 104b, and each functional unit is communicatively connected via a bus 105b. The hardware configuration shown in FIG. 1B is similar to the hardware configuration shown in FIG. 1A described above, except for the symbols (a, b) at the end, because the same numbers are used as in FIG. 1A, a duplicated description will be omitted.

[0027] In addition, the numerical values, processing timing, processing order, processing subject, and data (information) destination / source / storage location used in the embodiments described below are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0028] FIG. 1C is a block diagram showing an example of the overall functional configuration of an information processing system 1 according to this embodiment, which has a user terminal 100 and a server 111.

[0029] 1C, the user terminal 100 is made up of a first user terminal 100-1, a second user terminal 100-2, and user terminals 100-3 to n (n≧3) not shown, which have the same hardware configuration and functional units but are used by different users. Hereinafter, the functional units of the user terminal 100 will be described using the first user terminal 100-1, and redundant descriptions of the functional units of the second user terminal 100-2 and the user terminals 100-3 to n not shown will be omitted.

[0030] The first user terminal 100-1 has an input / output unit 112-1 and a transmission / reception unit (user terminal) 113-1, and the server 111 of this embodiment has a transmission / reception unit (server) 114, an association determination unit 115, an audio data analysis unit 116, and a volume adjustment unit 117.

[0031] 1C may be implemented as hardware or software (computer program). In the latter case, an information processing device capable of executing the computer program is applicable to the user terminal 100 and the server 111.

[0032] The input / output unit 112-1 receives data input from external devices (HMD 106-1, microphone 107-1, speaker 108-1, controller 109-1, etc.) connected to the first user terminal 100-1, and outputs data to these external devices.

[0033] Transmitting / receiving unit (user terminal) 113-1 transmits data held by first user terminal 100-1 (e.g., voice data generated by microphone 107-1, the position of the avatar of the user wearing the device, and the movement of the hands) to server 111 via network 110. Transmitting / receiving unit (user terminal) 113-1 also receives data transmitted from server 111 via network 110.

[0034] The transmitting / receiving unit (server) 114 (audio data transmitting means) transmits data held by the server 111 (for example, audio data with each avatar in the virtual space as a sound source, its position, hand movements, etc.) to the user terminal 100 as the destination via the network 110. The transmitting / receiving unit (server) 114 also receives data transmitted via the network 110 from the user terminal 100 as the source.

[0035] With this configuration, the server 111 and the user terminal 100 share audio data generated by each avatar as a sound source, information on the position of each avatar in the virtual space, and information on the hand movements of each avatar.

[0036] The association determination unit 115 (association determination means) determines whether or not the voice data transmitted from the first user terminal 100-1 is associated with an avatar linked to the second user terminal 100-2.

[0037] When association determination section 115 determines that there is association, voice data analysis section 116 (voice data analysis means) analyzes the voice data transmitted to second user terminal 100-2.

[0038] Volume adjustment section 117 (volume adjustment means) adjusts the volume of the audio data transmitted to second user terminal 100-2, which is output from speaker 108-2.

[0039] Below, with reference to Figures 2 and 3A to 3C, we will explain the process of adjusting the volume of audio data in accordance with the situation in the virtual space, which is executed in the first user terminal 100-1, the second user terminal 100-2, and the server 111 in this embodiment.

[0040] First, an example of a process for adjusting the volume of audio data in accordance with a situation in a virtual space according to this embodiment, which is executed in the information processing system 1, will be described with reference to the flowchart of FIG.

[0041] In FIG. 2, steps S201 and S202 are executed in first user terminal 100-1, steps S203 to S205 are executed in server 111, and steps S206 and S207 are executed in second user terminal 100-2.

[0042] In step S201, input / output unit 112-1 accepts a voice uttered by a user toward microphone 107-1 as first voice data.

[0043] In step S202, transmitting / receiving section (user terminal) 113-1 transmits the first voice data accepted in step S201 to server 111 via network 110, and this process on the side of first user terminal 100-1 ends.

[0044] In step S203, the transmitting / receiving unit (server) 114 receives, via the network 110, the first voice data transmitted in step S202.

[0045] In step S204, the relatedness determination unit 115, the voice data analysis unit 116, and the volume adjustment unit 117 execute a volume adjustment process to adjust the volume of the voice data transmitted to the second user terminal 100-2. The details of the volume adjustment process will be described later with reference to FIG. 3A.

[0046] In step S205, transmitting / receiving unit (server) 114 transmits the voice data whose volume has been adjusted in step S204 to second user terminal 100-2 via network 110, and this process on the server 111 side ends.

[0047] In step S206, transmission / reception unit (user terminal) 113-2 receives the voice data transmitted from server 111 via network 110 in step S205.

[0048] In step S207, input / output unit 112-2 outputs the voice data received in step S206 from speaker 108-2 at the volume adjusted in step S204, and this process on the side of second user terminal 100-2 ends.

[0049] Next, an example of the volume adjustment process in step S204 of FIG. 2 executed by the server 111 will be described with reference to the flowchart of FIG. 3A.

[0050] In Fig. 3A, first, in step S301, the association determination unit 115 executes an association determination process to determine whether or not the first voice data received from the first user terminal 100-1 in step S203 is associated with a second avatar linked to the second user terminal 100-2. Details of the association determination process will be described later with reference to Fig. 3B. The second avatar refers to the avatar of the user wearing the HMD 106-2. In other words, the first voice data being associated with the second avatar means that the user of the first user terminal 100-1 is talking to the user of the second user terminal 100-2.

[0051] In step S302, if the result of the association determination process in step S301 is that the first audio data is associated with the second avatar, the association determination unit 115 proceeds to step S303, and if not, terminates this process (volume adjustment process).

[0052] In step S303, the voice data analysis unit 116 executes a voice data analysis process to analyze the first voice data and the second voice data excluding the first voice data among the voice data transmitted to the second user terminal 100-2. The details of the voice data analysis process will be described later with reference to FIG. 3C. Note that the second voice data refers to all other voice data excluding the first voice data, such as the singing voices and performances of the performers and the cheers of other audience members, using the user terminals 100-3 to 100-n (not shown), when a live music event is held in a virtual space, for example.

[0053] In step S304, if the result of the voice data analysis process in step S303 is that the second voice data interferes with the first voice data, the voice data analysis unit 116 proceeds to step S305. On the other hand, if not, this process (volume adjustment process) ends. The second voice data interfering with the first voice data means a situation in which the first voice data cannot be heard or is very difficult to hear due to the second voice data.

[0054] In step S305, the volume adjustment unit 117 adjusts the volume of the voice data transmitted to the second user terminal 100-2, and ends this process (volume adjustment process). The volume is adjusted by increasing the volume of the first voice data or decreasing the volume of the second voice data according to the settings of the server 111 or the second user terminal 100-2. In this way, when the second voice data interferes with the first voice data, the volume of the first voice data is increased so that the first voice data becomes audible. Alternatively, when the second voice data interferes with the first voice data, the volume of the second voice data is decreased so that the first voice data becomes relatively audible.

[0055] Next, an example of the relatedness determination process in step S301 of FIG. 3A executed by server 111 will be described with reference to the flowchart of FIG. 3B.

[0056] In FIG. 3B, first, in step S306, the association determination unit 115 determines whether the second avatar holds his / her hand on his / her ear toward the first avatar (the avatar of the user wearing the HMD 106-1) which is the sound source of the first voice data. If the second avatar holds his / her hand on his / her ear toward the first avatar (YES in step S306), the process proceeds to step S307, and if not (NO in step S306), the process proceeds to step S308. The association determination unit 115 determines whether the second avatar holds his / her hand on his / her ear toward the first avatar based on the position of the second avatar set by the user input to the controller 109-2 and the movement of the hand of the user wearing the HMD 106-2 detected by the HMD 106-2. Here, the state where the second avatar holds his / her hand on his / her ear means the state where the user wearing the HMD 106-2 holds his / her hand on his / her ear. That is, even if the second avatar itself does not have an organ corresponding to an ear, if the hand is positioned on the side of the second avatar's face, the association determination unit 115 determines that the second avatar has a hand on its ear. In other words, since the gesture of placing a hand on an ear is a gesture performed in real space to make it easier to hear a sound from a specific direction, if the specific direction is the direction of the first avatar, it is determined that the first audio data is associated with the second avatar.

[0057] In step S307, relatedness determination unit 115 determines that the first voice data is related to the second avatar, and ends this process (relatedness determination process).

[0058] In step S308, association determination unit 115 determines whether or not the first avatar and the second avatar are in a friendly relationship. If they are in a friendly relationship (YES in step S308), the process proceeds to step S307, and if they are not in a friendly relationship (NO in step S308), the process proceeds to step S309. An example of a friendly relationship is when the first avatar and the second avatar are registered with server 111 as friends. Details of this determination method will be described later with reference to FIG. 5.

[0059] In step S309, relevance determination unit 115 determines whether or not the first avatar faces the direction of the second avatar. If the first avatar faces the direction of the second avatar (YES in step S309), the process proceeds to step S307, and if not (NO in step S309), the process proceeds to step S310. An example of a case in which the first avatar faces the direction of the second avatar is a case in which the second avatar exists on an extension line in front of the line of sight or face of the first avatar. As described above, whether or not the second avatar exists on an extension line in front of the face of the first avatar is determined based on the positions of the first and second avatars set by user operations on controllers 109-1 and 109-2, respectively.

[0060] In step S310, the relevance determination unit 115 determines whether or not the first voice data includes the name of the second avatar. Specifically, the relevance determination unit 115 performs voice recognition processing and natural language processing on the first voice data, and determines whether or not the name "Mr. B" displayed above the head of the second avatar, which will be described later in FIG. 4, is included in the first voice data. If the name of the second avatar is included (YES in step S310), the process proceeds to step S307; if not (NO in step S310), the process proceeds to step S311.

[0061] In step S311, relatedness determination unit 115 determines that the first voice data is not related to the second avatar, and ends this process (relatedness determination process).

[0062] Next, an example of the voice data analysis process in step S303 of FIG. 3A executed by the server 111 will be described with reference to the flowchart of FIG. 3C.

[0063] 3C, first, in step S312, the audio data analysis unit 116 determines whether the volume of the second audio data is greater than the volume of the first audio data. If the volume of the second audio data is greater than the volume of the first audio data (YES in step S312), the process proceeds to step S313, and if not (NO in step S312), the process proceeds to step S314. The volume of the audio data is compared using, for example, a decibel level.

[0064] In step S313, the voice data analysis unit 116 determines that the second voice data interferes with the first voice data, and ends this process (voice data analysis process).

[0065] In step S314, the voice data analysis unit 116 determines that the second voice data does not interfere with the first voice data, and ends this process (voice data analysis process).

[0066] Next, specific examples of the processes shown in the flowcharts of FIG. 2 and FIGS. 3A to 3C will be described with reference to FIG. 4 and FIG.

[0067] Fig. 4 is a diagram showing an example of a scene in a virtual space by VR provided by the server 111. Fig. 5 is a table showing an example of a friend registration status list, which is data indicating the presence or absence of friendly relationships between avatars, registered in advance as data by the server 111. Here, in this embodiment, the friend registration status list is registered in advance in ROM 102b or HDD 104b, but is not limited to this as long as CPU 101b (acquisition means) can acquire the friend registration status list during the determination process of step S308.

[0068] FIG. 4(a) shows a front view of a first avatar 401 and a second avatar 402 participating as spectators in a live music event held in a virtual space.

[0069] 4(a), first avatar 401 and second avatar 402 are side-by-side, and in front of them is a stage for a live music event in progress, with the surroundings being noisy with the voices of performers singing and playing, the cheers of other spectators, etc. As described above, first avatar 401 is an avatar operated by a user of first user terminal 100-1, and second avatar 402 is an avatar operated by a user of second user terminal 100-2.

[0070] 4(a), the name 403 of the first avatar 401, "Mr. A," is displayed above the head of the first avatar 401 so that it can be seen by the other avatars. Similarly, the name 404 of the second avatar 402, "Mr. B," is displayed above the head of the second avatar 402 so that it can be seen by the other avatars.

[0071] FIG. 4(b) shows a scene in which, from the state shown in FIG. 4(a), the user of first user terminal 100-1 speaks into microphone 107-1, processing proceeds from step S201 to step S203, and the relatedness determination processing (FIG. 3B) is about to be executed from step S306.

[0072] A dashed speech bubble 405 indicates that voice data corresponding to the voice "..." uttered by the user of the first user terminal 100-1 is generated from the first avatar 401 as a sound source. The dashed speech bubble 405 is displayed in FIG. 4(b) only for the purpose of explanation, and is not displayed in the virtual space. A hand 406 is the right hand of the second avatar 402.

[0073] In step S306, relatedness determination unit 115 determines whether second avatar 402 has its hand on its ear facing first avatar 401, which is the sound source of the first voice data. In Fig. 4(b), hand 406 of second avatar 402 is located on the right side of the face of second avatar 402, and first avatar 401 is located to the right of second avatar 402. Therefore, relatedness determination unit 115 determines that second avatar 402 has its hand on its ear facing first avatar 401, and proceeds to step S307.

[0074] FIG. 4(c) shows a scene in which, from the state shown in FIG. 4(a), the user of first user terminal 100-1 speaks into microphone 107-1, the process proceeds from step S201 to step S203, and the relatedness determination process (FIG. 3B) is about to be executed from step S306.

[0075] In step S306, relatedness determination unit 115 determines whether second avatar 402 holds his hand to his ear toward first avatar 401, which is the sound source of the first voice data. In Fig. 4(c), second avatar 402 has hand 406 (not shown) lowered, so relatedness determination unit 115 determines that second avatar 402 does not hold his hand to his ear toward first avatar 401, and proceeds to step S308.

[0076] In step S308, association determination unit 115 determines whether or not first avatar 401 and second avatar 402 are in a friendly relationship. In Fig. 5, the name of the avatar representing first avatar 401, "Mr. A", and the name of the avatar representing second avatar 402, "Mr. B", are registered as friends of each other. Therefore, association determination unit 115 determines that first avatar 401 and second avatar 402 are in a friendly relationship, and proceeds to step S307.

[0077] FIG. 4(d) shows a scene in which, from the state shown in FIG. 4(a), the user of first user terminal 100-1 speaks into microphone 107-1, the process proceeds from step S201 to step S203, and the relatedness determination process (FIG. 3B) is about to be executed from step S306.

[0078] In step S306, relatedness determination unit 115 determines whether second avatar 402 holds his / her hand against his / her ear toward first avatar 401, which is the sound source of the first voice data. In Fig. 4(d), second avatar 402 has hand 406 (not shown) down, similar to Fig. 4(c), and therefore proceeds to step S308.

[0079] In step S308, the association determination unit 115 determines whether the first avatar 401 and the second avatar 402 are in a friendly relationship. However, here, a friend registration status different from the friend registration status between the avatars shown in FIG. 5 is registered in the server 111. Specifically, in this example, the friend registration status indicates that the name of the avatar representing the first avatar 401, "Mr. A," and the name of the avatar representing the second avatar 402, "Mr. B," are not registered as friends of each other. Therefore, the association determination unit 115 determines that the first avatar 401 and the second avatar 402 are in a friendly relationship, and the process proceeds to step S307. Therefore, the association determination unit 115 determines that the first avatar 401 and the second avatar 402 are not in a friendly relationship, and the process proceeds to step S309.

[0080] In step S309, relatedness determination unit 115 determines whether or not first avatar 401 faces the direction of second avatar 402. Since second avatar 402 is present in front of first avatar 401 in Fig. 4(d), that is, on an extension of the line of sight of first avatar 401, relatedness determination unit 115 determines that first avatar 401 faces the direction of second avatar 402, and proceeds to step S307.

[0081] FIG. 4(e) shows a scene in which, from the state shown in FIG. 4(a), the user of first user terminal 100-1 speaks into microphone 107-1, processing proceeds from step S201 to step S203, and the relatedness determination processing (FIG. 3B) is about to be executed from step S306.

[0082] In step S306, relatedness determination unit 115 determines whether second avatar 402 holds his / her hand against his / her ear toward first avatar 401, which is the sound source of the first voice data. In Fig. 4(d), second avatar 402 has hand 406 (not shown) down, similar to Fig. 4(c), and therefore proceeds to step S308.

[0083] In step S308, the association determination unit 115 determines whether the first avatar 401 and the second avatar 402 are in a friendly relationship. However, here, a friend registration status different from the friend registration status between the avatars shown in FIG. 5 is registered in the server 111. Specifically, in this example, the friend registration status indicates that the name of the avatar representing the first avatar 401, "Mr. A," and the name of the avatar representing the second avatar 402, "Mr. B," are not registered as friends of each other. Therefore, the association determination unit 115 determines that the first avatar 401 and the second avatar 402 are in a friendly relationship, and the process proceeds to step S307. Therefore, the association determination unit 115 determines that the first avatar 401 and the second avatar 402 are not in a friendly relationship, and the process proceeds to step S309.

[0084] In step S309, relatedness determination unit 115 determines whether or not first avatar 401 faces the direction of second avatar 402. Since second avatar 402 is not in front of first avatar 401, i.e., on an extension of the line of sight in Fig. 4(e), relatedness determination unit 115 determines that first avatar 401 does not face the direction of second avatar 402, and proceeds to step S310.

[0085] In step S310, relatedness determination unit 115 determines whether or not first voice data having first avatar 401 as a sound source includes name 404 of second avatar 402. As shown in Fig. 4(e) , name 404 of second avatar 402, "Mr. B", is included in the first voice data, so the process proceeds to step S307.

[0086] In this manner, according to the present embodiment, when it is determined that the first voice data is related to the second avatar and the second voice data is analyzed to disturb the first voice data, the volume of the voice data transmitted to the second user terminal 100-2 is adjusted. This allows user A to have a conversation with user B, who is the target person, even in a noisy situation in the virtual space.

[0087] In the above description, the adjustment of the volume of the voice data transmitted to the second user terminal 100-2 is automatically performed by the server 111, and the user does not know whether the volume has been adjusted or not. However, the volume adjustment unit 117 may store that the volume of the voice data transmitted to the second user terminal 100-2 has been adjusted, and may notify the first user terminal 100-1 and the second user terminal 100-2 that the volume has been adjusted. This allows the user of the first user terminal 100-1 to know that the volume of his / her voice has been adjusted so that it is easier for the user of the second user terminal 100-2 to hear. Furthermore, when the volume of the second voice data is reduced in step S305, the user of the second user terminal 100-2 can know the reason why the volume of the second voice data is reduced.

[0088] In the above description, when the first voice data is determined to be related to the second avatar and the second voice data is analyzed to disturb the first voice data, the volume of the voice data transmitted to the second user terminal 100-2 is always adjusted. However, the volume adjustment unit 117 may not automatically adjust the volume, but may send a query to the user of the second user terminal 100-2 as to whether or not to adjust the volume, and allow the user to select whether or not to adjust the volume. In this case, in step S305, the volume adjustment unit 117 does not adjust the volume of the voice data transmitted to the second user terminal 100-2, but stores information that the volume should be adjusted and the adjustment amount of the voice data. After that, the voice data with the volume not adjusted and a query to ask the user to select whether or not to adjust the volume are transmitted to the second user terminal 100-2 are transmitted. In response to this inquiry, input / output unit 112-2 of second user terminal 100-2 causes HMD 106-2 to display a dialog prompting the user to select whether or not to adjust the volume, allowing the user to select whether or not to adjust the volume with controller 109-2. If the user chooses to select the volume, a volume adjustment request is sent to server 111, and volume adjustment unit 117 adjusts the volume of the audio data by the stored adjustment amount and sends it to second user terminal 100-2.

[0089] In the above description, steps S306 to S310 of the association determination process are determined in the order of the steps, but they may be determined simultaneously, and an "OR condition" may be used in which the process proceeds to step S307 if the determination is satisfied in any of the steps. Alternatively, an "AND condition" may be used in which the process proceeds to step S307 if any of the steps are satisfied simultaneously. For example, if the AND condition is set for steps S308 and S309, the process may proceed to step S307 if the first avatar and the second avatar are in a friendly relationship and the first avatar is facing the direction of the second avatar.

[0090] In the above description, the server 111 adjusts the volume of the audio data transmitted to the second user terminal 100-2, but the volume may be adjusted by the second user terminal 100-2. In this case, the server 111 transmits to the second user terminal 100-2 audio data whose volume has not been adjusted and the amount of volume adjustment for each piece of audio data.

[0091] In the above explanation, a client-server system in which the server 111 exists has been described, but a peer-to-peer system may also be used. In that case, the server 111 does not exist, and the functional configuration and processing of the server 111 are performed by the user terminal 100 instead.

[0092] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0093] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0094] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0095] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An information processing system that provides a virtual space including avatars linked to each of the plurality of user terminals, the information processing system including three or more user terminals connected via a network, each having an audio input unit and an audio output unit, the information processing system including: an audio generation unit that generates audio data using each avatar linked to each of the plurality of user terminals as a sound source based on audio input to the audio input unit of each of the plurality of user terminals; an audio data transmission unit that transmits the audio data using each avatar as a sound source to the plurality of user terminals; and a first audio data using a first avatar linked to a first user terminal of the plurality of user terminals as a sound source to a second user terminal of the plurality of user terminals. an association determination means for determining whether the first voice data is associated with a second avatar linked to the first avatar; an audio data analysis means for analyzing the first voice data and second voice data excluding the first voice data from the voice data transmitted to the second user terminal by the audio data transmitting means; and a volume adjustment means for adjusting a volume of the voice data transmitted to the second user terminal, the volume being output by the audio output unit of the second user terminal, when the association determination means determines that the first voice data is associated with the second avatar and the audio data analysis means analyzes that the second voice data interferes with the first voice data. (Configuration 2) The information processing system according to configuration 1, wherein the volume adjustment means increases the volume of the first voice data among the voice data transmitted to the second user terminal. (Configuration 3) The information processing system according to configuration 1, wherein the volume adjustment means reduces the volume of the second voice data among the voice data transmitted to the second user terminal. (Configuration 4) An information processing system described in any one of configurations 1 to 3, characterized in that the association determination means determines that the first voice data is associated with the second avatar when the second avatar is holding its hand to its ear toward the first avatar. (Configuration 5) The multiple user terminals further have a first detection unit that detects hand movements of the users using each of the multiple user terminals, and a setting unit that sets the position of the avatars associated with each of the multiple user terminals in the virtual space in accordance with user operation, and the information processing system further has an avatar action control means that, when the hand movement is detected by the first detection unit of the second user terminal, moves the hand of the second avatar in accordance with the detection result, and the association determination means determines whether the second avatar is holding its hand to its ear toward the first avatar in accordance with the set positions of the first and second avatars and the hand movement of the second avatar controlled by the avatar action control means. (Configuration 6) An information processing system described in any one of configurations 1 to 5, characterized in that the association determination means determines that the first voice data is associated with the second avatar when the first avatar and the second avatar are in a friendly relationship. (Configuration 7) The information processing system described in configuration 6, further comprising an acquisition means for acquiring data indicating whether or not there is a friendly relationship between avatars linked to each of the multiple user terminals, wherein the association determination means determines whether or not the first avatar and the second avatar are in a friendly relationship based on the data. (Configuration 8) An information processing system described in any one of configurations 1 to 7, characterized in that the association determination means determines that the first voice data is associated with the second avatar when the first avatar is facing in the direction of the second avatar. (Configuration 9) An information processing system according to configuration 8, wherein the association determination means determines that the first avatar is facing toward the second avatar when the second avatar is present on an extension line in front of the face of the first avatar. (Configuration 10) The information processing system described in Configuration 9, characterized in that the multiple user terminals further have a setting unit that sets the position of the avatar associated with each of them in the virtual space in accordance with user operation, and the association determination means determines whether the second avatar is present on an extension line in front of the face of the first avatar in accordance with the set positions of the first and second avatars. (Configuration 11) An information processing system according to configuration 8, wherein the association determination means determines that the first avatar is facing in the direction of the second avatar when the second avatar is present in the line of sight of the first avatar. (Configuration 12) The information processing system described in Configuration 11, wherein the multiple user terminals further have a setting unit that sets the position of the avatar associated with each of them in the virtual space in accordance with user operation, and a second detection unit that detects the line of sight of the user using each of them, and the information processing system further includes a gaze direction setting means that sets the gaze direction of the first avatar in accordance with the set position of the first avatar and the line of sight detected by the second detection unit of the first user terminal, and the association determination means determines whether the second avatar is present in the gaze direction of the set first avatar in accordance with the set position of the second avatar. (Configuration 13) An information processing system described in any one of configurations 1 to 12, characterized in that the association determination means determines that the first voice data is associated with the second avatar when the first voice data contains the name of the second avatar. (Configuration 14) An information processing system described in any one of configurations 1 to 13, characterized in that the audio data analysis means analyzes that the second audio data interferes with the first audio data when the volume of the second audio data is greater than the volume of the first audio data. (Configuration 15) An information processing system as described in any one of configurations 1 to 14, characterized in that the volume adjustment means stores that the volume of the audio data to be transmitted to the second user terminal has been adjusted, and notifies each of the first and second user terminals that the volume has been adjusted. (Configuration 16) The information processing system described in any one of configurations 1 to 15, characterized in that when the association determination means determines that the first voice data is associated with the second avatar and the voice data analysis means analyzes that the second voice data interferes with the first voice data, the volume adjustment means makes an inquiry to a second user terminal as to whether or not to adjust the volume, and adjusts the volume when a user selection to adjust the volume is made on the second user terminal in response to the inquiry. (Configuration 17) The information processing system according to any one of configurations 1 to 16, wherein the plurality of user terminals each have an HMD worn by a user who uses the terminal. (Configuration 18) An edge device that functions as one of the plurality of user terminals provided in the information processing system according to any one of configurations 1 to 17, the edge device comprising the volume control means. (Configuration 19) A server provided in an information processing system according to any one of configurations 1 to 17, connected to the plurality of user terminals via the network, characterized in that the server comprises a voice data transmission means, a relatedness determination means, and a voice data analysis means. (Method 1) A control method for an information processing system that provides a virtual space including avatars linked to each of the plurality of user terminals, the information processing system including three or more user terminals connected via a network, each having an audio input unit and an audio output unit, the control method including a voice generation step of generating voice data using each avatar linked to each of the plurality of user terminals as a sound source based on a voice input to the audio input unit of each of the plurality of user terminals, a voice data transmission step of transmitting the voice data using each avatar as a sound source to the plurality of user terminals, and a first voice data using a first avatar linked to a first user terminal of the plurality of user terminals as a sound source to a second user terminal of the plurality of user terminals. a voice data analysis step of analyzing the first voice data and second voice data excluding the first voice data from the voice data transmitted to the second user terminal in the voice data transmitting step; and a volume adjustment step of adjusting a volume of the voice data transmitted to the second user terminal, which is output from the voice output unit of the second user terminal, when the first voice data is determined to be associated with the second avatar in the association determination step and the second voice data is analyzed to interfere with the first voice data in the voice data analysis step. (Program 1) A program for causing a computer to function as the edge device described in configuration 18 by loading the program into the computer and executing the program. (Program 2) A program for causing a computer to function as the server described in configuration 19 by loading and executing the program into the computer. (Storage medium 1) A computer-readable storage medium storing the program described in configuration 21. (Storage medium 2) A computer-readable storage medium storing the program described in configuration 22. [Explanation of symbols]

[0096] 1. Information Processing Systems 100 user terminals 106 HMD 107 Mike 108 Speaker 110 Network 112 Input / output section 113 Transmitter / receiver (user terminal) 114 Transmitting / receiving unit (server) 115 Related Judgment Section 116 Voice Data Analysis Department 117 Volume control

Claims

1. An information processing system that provides a virtual space including three or more user terminals, each having an audio input unit and an audio output unit, connected via a network, and an avatar linked to each of the user terminals, a voice generating means for generating voice data using each avatar associated with each of the plurality of user terminals as a sound source based on a voice input to the voice input unit of each of the plurality of user terminals; a voice data transmitting means for transmitting voice data having each of the avatars as a sound source to the plurality of user terminals; an association determination means for determining whether or not first voice data, the first voice data having a first avatar associated with a first user terminal among the plurality of user terminals as a sound source, is associated with a second avatar associated with a second user terminal among the plurality of user terminals; a voice data analysis means for analyzing the first voice data and second voice data excluding the first voice data, among the voice data transmitted to the second user terminal by the voice data transmission means; a volume adjustment means for adjusting a volume of the voice data transmitted to the second user terminal, the volume being output by the voice output unit of the second user terminal, when the association determination means determines that the first voice data is associated with the second avatar and the voice data analysis means analyzes that the second voice data interferes with the first voice data; An information processing system comprising:

2. 2. The information processing system according to claim 1, wherein the volume adjustment means increases the volume of the first voice data among the voice data transmitted to the second user terminal.

3. 2. The information processing system according to claim 1, wherein the volume adjustment means reduces the volume of the second voice data among the voice data transmitted to the second user terminal.

4. The information processing system according to claim 1, characterized in that the association determination means determines that the first voice data is associated with the second avatar when the second avatar is holding its hand to its ear and facing the first avatar.

5. The plurality of user terminals include A first detection unit that detects hand movements of a user who uses each of the devices; a setting unit that sets the position of each associated avatar in the virtual space in response to a user operation; and The information processing system includes: further comprising an avatar action control means for, when the hand movement is detected by the first detection unit of the second user terminal, moving a hand of the second avatar in response to the detection result; The information processing system according to claim 4, characterized in that the association determination means determines whether the second avatar is holding its hand to its ear toward the first avatar based on the set positions of the first and second avatars and the hand movement of the second avatar controlled by the avatar action control means.

6. The information processing system according to claim 1, characterized in that the association determination means determines that the first voice data is associated with the second avatar when the first avatar and the second avatar are in a friendly relationship.

7. The method further includes acquiring data indicating whether or not there is a friendly relationship between the avatars associated with the plurality of user terminals, 7. The information processing system according to claim 6, wherein said relation determining means determines whether or not said first avatar and said second avatar are in a friendly relationship based on said data.

8. The information processing system according to claim 1, characterized in that the association determination means determines that the first voice data is associated with the second avatar when the first avatar is facing in the direction of the second avatar.

9. The information processing system according to claim 8 , wherein the association determination means determines that the first avatar is facing toward the second avatar when the second avatar is present on an extension line in front of the face of the first avatar.

10. The plurality of user terminals further include a setting unit that sets a position of an avatar associated with each of the user terminals in the virtual space in response to a user operation, The information processing system according to claim 9, characterized in that the association determination means determines whether or not the second avatar is present on an extension line in front of the face of the first avatar, depending on the set positions of the first and second avatars.

11. The information processing system according to claim 8 , wherein the association determination means determines that the first avatar is facing in the direction of the second avatar when the second avatar is present in a line of sight of the first avatar.

12. The plurality of user terminals include a setting unit that sets the position of each associated avatar in the virtual space in response to a user operation; A second detection unit that detects the line of sight of a user who uses each of the devices; Further comprising: The information processing system includes: a gaze direction setting means for setting a gaze direction of the first avatar in accordance with the set position of the first avatar and the gaze detected by the second detection unit of the first user terminal, The information processing system according to claim 11, characterized in that the association determination means determines whether the second avatar is present in the line of sight of the set first avatar depending on the position of the set second avatar.

13. The information processing system according to claim 1, characterized in that the association determination means determines that the first voice data is associated with the second avatar when the first voice data contains the name of the second avatar.

14. The information processing system according to claim 1, characterized in that the audio data analysis means analyzes that the second audio data interferes with the first audio data when the volume of the second audio data is greater than the volume of the first audio data.

15. The information processing system according to claim 1, characterized in that the volume adjustment means stores that the volume of the audio data transmitted to the second user terminal has been adjusted, and notifies each of the first and second user terminals that the volume has been adjusted.

16. The volume control means is when the association determination means determines that the first voice data is associated with the second avatar and the voice data analysis means analyzes that the second voice data interferes with the first voice data, an inquiry is made to a second user terminal as to whether or not to adjust the volume; 2. The information processing system according to claim 1, wherein the volume is adjusted when a user selection to adjust the volume is made at the second user terminal in response to the inquiry.

17. 2. The information processing system according to claim 1, wherein each of the plurality of user terminals has an HMD worn by a user who uses the terminal.

18. An edge device that functions as one of the plurality of user terminals included in the information processing system according to claim 1, An edge device comprising the volume control means.

19. 2. A server connected to the plurality of user terminals via the network, the server being included in the information processing system according to claim 1, A server comprising the voice data transmitting means, the relevance determining means, and the voice data analyzing means.

20. A method for controlling an information processing system that provides a virtual space including avatars linked to each of three or more user terminals connected via a network, each of the user terminals having an audio input unit and an audio output unit, the method comprising: a voice generating step of generating voice data using each avatar associated with each of the plurality of user terminals as a sound source based on the voice input to the voice input unit of each of the plurality of user terminals; a voice data transmitting step of transmitting voice data having each of the avatars as a sound source to the plurality of user terminals; an association determination step of determining whether or not first voice data, the first voice data having a first avatar associated with a first user terminal among the plurality of user terminals as a sound source, is associated with a second avatar associated with a second user terminal among the plurality of user terminals; a voice data analysis step of analyzing the first voice data and second voice data excluding the first voice data, among the voice data transmitted to the second user terminal in the voice data transmission step; a volume adjustment step of adjusting a volume of the voice data transmitted to the second user terminal, which is output from the voice output unit of the second user terminal, when the first voice data is determined to be associated with the second avatar in the association determination step and the second voice data is analyzed to interfere with the first voice data in the voice data analysis step; A control method comprising:

21. A program for causing a computer to function as the edge device according to claim 18 by reading the program into a computer and executing the program.

22. 20. A program for causing a computer to function as the server according to claim 19 by reading and executing the program into a computer.

23. A computer-readable storage medium storing the program according to claim 21.

24. A computer-readable storage medium storing the program according to claim 22.

Images