Session management device

The session management device addresses audio overlap issues in multi-user sessions by synchronizing audio outputs and inputs based on terminal proximity, improving user experience and clarity in virtual spaces.

JP2024168005APending Publication Date: 2024-12-05NTT DOCOMO INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023084378
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

When multiple users at the same location participate in sessions using personal terminals, audio input and output from these terminals often overlap, making it difficult for participants to hear clearly due to asynchronous voice outputs and microphone interference.

Method used

A session management device that manages sessions between terminals by acquiring location information and controlling audio processing to synchronize audio outputs and inputs based on terminal proximity, using methods like short-range wireless communication and audio data synchronization.

Benefits of technology

Improves convenience and clarity in sessions by reducing audio overlap and ensuring synchronized audio output among terminals located at the same base, enhancing the overall user experience in virtual spaces.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168005000001_ABST
    Figure 2024168005000001_ABST
Patent Text Reader

Abstract

To improve convenience when a plurality of users present in the same site participate in a session by using individual terminals.SOLUTION: A server 10 manages a session between a plurality of user terminals 20 including a user terminal 20A and a user terminal 20B. A position acquisition unit 151 acquires position information of the user terminal 20A and position information of the user terminal 20B. A voice control unit 152 controls processing of voice data in the user terminal 20A and user terminal 20B on the basis of whether the user terminal 20A and user terminal 20B are located in the same site B1.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a session management device. [Background technology]

[0002] In recent years, opportunities for communication via a network have been increasing. Conventionally, in many cases, one terminal is installed at one location, and when multiple users are at the same location, they participate in a session by sharing one terminal. For example, the following Patent Document 1 discloses an online conference space construction system that is arranged in a shared office space where multiple workers work. The online conference space construction system has a movable chair and a panel that shields a part of the chair in the horizontal direction. The online conference space construction system also includes a plurality of panel-equipped chairs, the panel of which is integrated with the chair so that it can move together with the chair, and a camera unit having a camera that photographs an office worker seated on at least one of the plurality of panel-equipped chairs. The online conference space construction system allows multiple panel-equipped chairs to be arranged so that a space for holding an online conference can be constructed by the panel in at least a part of the shared office space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-056245 A Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, even when multiple users are at the same location, there is a need to participate in a session using individual terminals. In this case, there is a problem that the voices input and output to the multiple terminals located at the same location overlap, which may cause a situation where the voices are difficult for the participants of the session to hear. Specifically, for example, the same voice is output from multiple terminals at slightly different times, making it difficult to hear the voice. In addition, for example, the microphones of multiple terminals may pick up the voice of the same user, making it difficult for the other party to hear the voice.

[0005] An object of the present invention is to improve convenience when a plurality of users at the same location participate in a session using individual terminals. [Means for solving the problem]

[0006] A session management device according to one embodiment of the present invention is a session management device that manages sessions between multiple terminals including a first terminal and a second terminal, and includes a location acquisition unit that acquires location information of the first terminal and location information of the second terminal, and an audio control unit that controls processing of audio data in the first terminal and the second terminal based on whether the first terminal and the second terminal are located at the same base. Effect of the Invention

[0007] According to one aspect of the present invention, it is possible to improve the convenience when a plurality of users at the same location participate in a session using individual terminals. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of a system 1 according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing the configuration of a user terminal 20. [Diagram 3] FIG. 2 is a block diagram showing the configuration of a server 10. [Figure 4A]FIG. 2 is a diagram illustrating an example of a virtual space image PC. [Figure 4B] FIG. 2 is a diagram illustrating an example of a virtual space image PC. [Figure 4C] FIG. 2 is a diagram illustrating an example of a virtual space image PC. [Diagram 5] FIG. 2 is a diagram showing a schematic arrangement of avatars A1 to A3 in a virtual space VS. [Figure 6] 4 is a flowchart showing the operation of the processing device 103. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A. First embodiment A-1. System configuration 1 is a block diagram showing a configuration of a system 1 according to an embodiment. The system 1 includes a server 10 and a plurality of user terminals 20. The server 10 and the user terminals 20A to 20C are connected via a network N.

[0010] The user terminal 20A is owned by a user U1. The user terminal 20B is owned by a user U2. The user terminal 20C is owned by a user U3. The user terminal 20A is an example of a first terminal, the user terminal 20B is an example of a second terminal, and the user terminal 20C is an example of a third terminal. Furthermore, the user U1 is an example of a first user, and the user U2 is an example of a second user. Hereinafter, when there is no need to distinguish between the multiple user terminals 20A to 20C, they will simply be referred to as "user terminal 20". Furthermore, when there is no need to distinguish between the multiple users U1 to U3, they will simply be referred to as "user U".

[0011] The system 1 enables the user U to have various experiences in the virtual space VS by displaying the virtual space VS (see, for example, FIGS. 4A to 4C) on the user terminal 20. In this embodiment, a case will be mainly described in which the user U converses with another user U in the virtual space VS.

[0012] A user U acts as an avatar A in the virtual space VS. The avatar A corresponding to user U1 is avatar A1, the avatar A corresponding to user U2 is avatar A2, and the avatar A corresponding to user U3 is avatar A3. Avatar A1 is an example of a first avatar, and avatar A2 is an example of a second avatar. When there is no need to distinguish between the multiple avatars A1 to A3, they will simply be referred to as "avatar A."

[0013] For example, an image of the virtual space VS (hereinafter referred to as "virtual space image PC") displayed on the user terminal 20A shows the field of view of an avatar A1 corresponding to the user U1. The viewpoint of the virtual space image PC may be positioned near the avatar A1 so that the avatar A1 appears in the virtual space image PC. By operating the avatar A1 using the user terminal 20A, the user U1 can move within the virtual space VS and communicate, including have conversations, with other users U2 and U3.

[0014] Figures 4A to 4C are diagrams showing an example of a virtual space image PC. Figures 4A to 4C show virtual space images PC1 to PC3 displayed on user terminals 20A to 20C when avatars A1 to A3 gather in one place and converse in the virtual space VS. In Figures 4A to 4C, text surrounded by dotted lines indicates that the text is being output as voice.

[0015] 5 is a diagram showing a schematic arrangement of avatars A1 to A3 in virtual space VS, showing the arrangement of avatars A1 to A3 as viewed from above. In this embodiment, avatar A1 and avatar A2 act together in virtual space VS. In the example of FIG. 5, avatar A1 and avatar A2 are positioned so as to face avatar A3. Of avatar A1 and avatar A2, avatar A1 is positioned closer to avatar A3, and avatar A2 is positioned behind avatar A1.

[0016] Virtual space image PC1 shown in FIG. 4A is an image displayed on user terminal 20A and is an image from the line of sight of avatar A1. The face of avatar A3 is shown in virtual space image PC1. Virtual space image PC2 shown in FIG. 4B is an image displayed on user terminal 20B and is an image from the line of sight of avatar A2. The face of avatar A3 and the back of avatar A1 are shown in virtual space image PC2. Virtual space image PC3 shown in FIG. 4C is an image displayed on user terminal 20C and is an image from the line of sight of avatar A3. The faces of avatar A1 and avatar A2 are shown in virtual space image PC3.

[0017] As shown in Figures 4A and 4B, user terminal 20A and user terminal 20B each display different images (user terminal 20A displays an image from the line of sight of avatar A1, and user terminal 20B displays an image from the line of sight of avatar A2), but the audio they output is the same (the speaking audio of avatar A3).

[0018] The server 10 is an example of a session management device that manages sessions between a plurality of user terminals 20 including a user terminal 20A and a user terminal 20B. In this embodiment, a session is transmission and reception of data that realizes a conversation between a plurality of avatars A including an avatar A1 corresponding to a user U1 and an avatar A2 corresponding to a user U2 in a virtual space VS.

[0019] The server 10 manages the virtual space VS. More specifically, the server 10 manages virtual space data VD (see FIG. 3) for causing the user terminal 20 to output the virtual space VS. Furthermore, the server 10 generates output data for outputting the virtual space VS at each user terminal 20 based on the virtual space data VD and transmission data transmitted from each user terminal 20, and transmits the output data to each user terminal 20. The output of the virtual space VS includes displaying an image of the virtual space VS and outputting sound generated in the virtual space VS. That is, the output data includes image data and sound data. The sound data includes, for example, sound indicating a conversation between users U (between avatars A).

[0020] The user terminal 20 is, for example, an information processing terminal such as a smartphone, a tablet terminal, a personal computer, or a head mounted display. Although three user terminals 20A to 20C are illustrated in Fig. 1, the number of user terminals 20 may be four or more, or may be two.

[0021] In this embodiment, the user terminal 20A and the user terminal 20B (i.e., the user U1 and the user U2) are located at the same base B1. The base B1 is an example of a first base. Being located at the same base corresponds to, for example, the user U1 and the user U2 being located in a place close enough to be able to talk directly without using a communication device or the like. Specifically, being in the same room at home, sitting at the same table in a conference room or restaurant, sitting at adjacent seats at work, and the like, corresponds to being located at the same base. Being located at the same base may also be expressed as the distance between the user terminal 20A and the user terminal 20B being equal to or less than a predetermined distance. The predetermined distance is, for example, a range in which short-distance wireless communication such as Bluetooth (registered trademark) can be connected, and may be a distance of about 10 m. In addition, the user terminal 20C (i.e., the user U3) is located at another base B2 different from the base B1. The base B2 is an example of a second base.

[0022] A-2. User terminal 20 2 is a block diagram showing the configuration of a user terminal 20. The user terminal 20 includes a communication device 201, an input device 202, a display device 203, a microphone 204, a speaker 205, an imaging device 206, a GPS (Global Positioning System) device 207, a storage device 208, a processing device 209, and a bus 220 that interconnects these devices.

[0023] The communication device 201 communicates with the server 10 using wireless communication or wired communication. In this embodiment, the communication device 201 has an interface connectable to a network N, and communicates with the communication device 101 of the server 10 (see FIG. 3) via the network N.

[0024] The input device 202 is an input device (e.g., a keyboard, a mouse, a switch, a button, or a sensor) that receives input from the outside. The display device 203 is a display device (e.g., various display panels such as a liquid crystal display panel or an organic EL display panel) that displays information to the outside. The display device 203 and the input device 202 may be integrated into one (e.g., a touch panel).

[0025] It should be noted that, in the case where the user terminal 20 is a head-mounted display, a projection device that projects an image onto lenses disposed in front of both eyes of the user U is provided instead of the display device 203.

[0026] The microphone 204 collects sounds generated around the user terminal 20 and outputs them as audio data. In this embodiment, the sounds collected by the microphone 204 are mainly the speech of the user U. The speaker 205 outputs sounds that can be heard around the user terminal 20 based on the input audio data. In this embodiment, the sounds output by the speaker 205 are mainly the speech of other users U. The speaker 205 may be a stereo speaker.

[0027] The imaging device 206 generates an image of the user U (hereinafter referred to as a "user image"). When the user terminal 20 is a smartphone, a tablet terminal, or a personal computer, the imaging device 206 is, for example, an in-camera whose imaging direction faces the user U. Also, when the user terminal 20 is, for example, a head-mounted display, the imaging device 206 is, for example, a camera for detecting a line of sight.

[0028] The GPS device 207 receives radio waves from multiple satellites and generates location information from the received radio waves. The location information indicates the location of the user terminal 20. The location information may be in any format as long as it can identify the location. In this embodiment, the location information is indicated by, for example, latitude and longitude. Note that the location information may be obtained by means other than the GPS device 207. For example, a beacon or the like installed in the facility may transmit information identifying the name of the facility or the room number in which the user terminal 20 is located.

[0029] The storage device 208 is a recording medium readable by the processing device 209. The storage device 208 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), and an Electrically Erasable Programmable Read Only Memory (EEPROM). The volatile memory is, for example, a Random Access Memory (RAM). The storage device 208 stores a program PG2. The program PG2 is a program for operating the user terminal 20.

[0030] The processing device 209 includes one or more central processing units (CPUs). The one or more CPUs are an example of one or more processors. Each of the processor and the CPU is an example of a computer.

[0031] The processing device 209 reads the program PG2 from the storage device 208. The processing device 209 executes the program PG2 to function as an input control unit 211 and an output control unit 212. At least some of the functions of the input control unit 211 and the output control unit 212 may be configured by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array).

[0032] The input control unit 211 transmits information input from the input device 202 (hereinafter referred to as “input information”) to the server 10. The input information is, for example, information instructing the movement direction of the avatar A in the virtual space VS, the spoken voice of the user U, or a captured image of the user U.

[0033] The output control unit 212 outputs information (hereinafter referred to as "output information") transmitted from the server 10 from the display device 203. The output information is, for example, a virtual space image PC or a speech voice of another user U (another avatar A).

[0034] A-3. Server 10 3 is a block diagram showing the configuration of the server 10. The server 10 includes a communication device 101, a storage device 102, a processing device 103, and a bus 160 that interconnects these devices.

[0035] The communication device 101 communicates with the user terminal 20 using wireless communication or wired communication. In this embodiment, the communication device 101 has an interface connectable to a network N, and communicates with a communication device 201 (see FIG. 2) of the user terminal 20 via the network N.

[0036] The storage device 102 is a recording medium readable by the processing device 103. The storage device 102 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM. The storage device 102 stores a program PG1 and virtual space data VD.

[0037] The program PG1 is a program for operating the server 10. The virtual space data VD is data for displaying the virtual space VS on the user terminal 20. The virtual space data VD includes, for example, information indicating the shapes and appearances of objects placed in the virtual space VS, information indicating the positions of the objects, and information indicating the shape and appearance of an avatar A corresponding to a user U.

[0038] The processing unit 103 includes one or more CPUs. The one or more CPUs are examples of one or more processors. Each of the processor and the CPU is an example of a computer.

[0039] The processing device 103 reads the program PG1 from the storage device 102. The processing device 103 executes the program PG1 to function as an avatar control unit 120, a virtual space control unit 130, a transmission data generation unit 140, and a terminal control unit 150. The terminal control unit 150 also includes a position acquisition unit 151 and a voice control unit 152. At least some of the functions of the avatar control unit 120, the virtual space control unit 130, the transmission data generation unit 140, and the terminal control unit 150 (the position acquisition unit 151 and the voice control unit 152) may be configured by circuits such as a DSP, an ASIC, a PLD, and an FPGA.

[0040] A-4. Details of the processing device 103 Next, the avatar control unit 120, the virtual space control unit 130, the transmission data generation unit 140, and the terminal control unit 150 (the position acquisition unit 151 and the audio control unit 152) realized by the processing device 103 will be described.

[0041] The avatar control unit 120 controls the behavior of the avatar A in the virtual space VS. For example, in the case of the user terminal 20A, the avatar control unit 120 determines the position of the avatar A1 in the virtual space VS based on information transmitted from the user terminal 20A indicating the moving direction of the avatar A1 in the virtual space VS. The avatar control unit 120 also determines the facial expression of the avatar A1 based on the captured image (the facial expression of the user U1) transmitted from the user terminal 20A. The avatar control unit 120 also controls the speech of the avatar A1 based on the speech of the user U1 transmitted from the user terminal 20A. The speech of the avatar A1 is basically the speech of the user U1 itself, but the data of the speech may be processed by, for example, a voice changer. The avatar control unit 120 also determines the movement of the mouth of the avatar A1 based on the speech of the user U1 transmitted from the user terminal 20A.

[0042] Virtual space control unit 130 controls virtual space VS. Controlling the virtual space VS includes reflecting the behavior of avatar A determined by avatar control unit 120 on the virtual space data VD. Controlling the virtual space VS also includes reflecting changes in the virtual space VS caused by, for example, the behavior of avatar A on the virtual space data VD.

[0043] The transmission data generation unit 140 generates transmission data to be transmitted to the user terminals 20A to 20C. The transmission data is generated individually for each of the user terminals 20A to 20C and transmitted to each of the user terminals 20A to 20C. The virtual space data VD mainly includes image data and audio data. The image data causes the display device 203 of the user terminal 20 to display the virtual space image PC. The audio data causes the speaker 205 of the user terminal 20 to output sound in the virtual space VS.

[0044] Taking the transmission data to be transmitted to the user terminal 20A as an example, the transmission data generation unit 140 identifies a portion of the virtual space VS included in the visual field of the avatar A1 based on the position (including the line of sight direction) of the avatar A1 in the virtual space VS. The transmission data generation unit 140 extracts image data showing the portion of the virtual space VS from the virtual space data VD. When other avatars A2 and A3 are located in the portion of the virtual space VS, the images of the other avatars A2 and A3 are also included in the image data. In addition, the transmission data generation unit 140 identifies a generated sound occurring around the position of the avatar A1 in the virtual space VS, and generates audio data showing the generated sound. In this embodiment, the audio data mainly includes the speech of the other avatars A with whom the avatar A1 (user U1) is conversing in the virtual space VS.

[0045] The terminal control unit 150 controls the input and output of voice in the user terminal 20. More specifically, when multiple user terminals 20 are located at the same base, such as user terminal 20A and user terminal 20B located at base B1, the terminal control unit 150 controls the input and output of voice in the multiple user terminals 20. In other words, when the multiple user terminals 20 are located at different bases, each user terminal 20 performs input and output of voice without being controlled by the terminal control unit 150.

[0046] The position acquisition unit 151 acquires the position information of the user terminal 20A and the position information of the user terminal 20B. The position acquisition unit 151 transmits a request for transmitting the position information generated by the GPS device 207 to the user terminal 20 (including the user terminal 20A and the user terminal 20B) that accesses the virtual space VS, for example. The user terminal 20 that receives the request for transmitting the position information transmits the position information generated by each GPS device 207 to the server 10. The acquisition of the position information is performed at predetermined intervals (for example, every several tens of seconds) while the user terminal 20 is accessing the virtual space VS, for example.

[0047] The voice control unit 152 controls the processing of voice data in the user terminal 20A and the user terminal 20B based on whether the user terminal 20A and the user terminal 20B are located at the same location B1. The voice control unit 152 judges whether the location information of the user terminal 20A and the location information of the user terminal 20B acquired by the location acquisition unit 151 match (not limited to a perfect match, including a case where the difference is equal to or less than a predetermined value). If the location information of the user terminal 20A and the location information of the user terminal 20B do not match, the voice control unit 152 judges that the user terminal 20A and the user terminal 20B are not located at the same location B1. In this case, each of the user terminals 20A and 20B inputs and outputs voice without being controlled by the terminal control unit 150 (voice control unit 152).

[0048] When the location information of the user terminal 20A and the location information of the user terminal 20B match, the voice control unit 152 causes at least one of the user terminal 20A and the user terminal 20B to execute a connection request for short-distance wireless communication such as BLE (Bluetooth (registered trademark) Low Energy) from the other. As described above, the same base corresponds to being located in a place close enough for the user U1 and the user U2 to talk directly to each other. This includes cases where the location information only matches and it cannot be said that the users are located at the same base B1, such as when they are in different rooms in the same building. For this reason, the voice control unit 152 judges whether the user terminals 20A and 20B are located at the same base B1 based on whether they can connect (pair) by short-distance wireless communication. Note that a pairing key may be transmitted from the voice control unit 152 to prevent the user terminals 20A and 20B from erroneously pairing with terminals other than the user terminal 20A and the user terminal 20B.

[0049] If the short-distance wireless connection is not established, the audio control unit 152 determines that the user terminal 20A and the user terminal 20B are not located at the same location B1. In this case, each of the user terminals 20A and 20B inputs and outputs audio without being controlled by the terminal control unit 150 (audio control unit 152).

[0050] On the other hand, if a short-distance wireless connection is established, the audio control unit 152 determines that the user terminal 20A and the user terminal 20B are located at the same location B1. The audio control unit 152 continues the pairing state between the user terminal 20A and the user terminal 20B, and controls the input and output of audio in the user terminal 20A and the user terminal 20B as follows.

[0051] [Audio output] When the user terminal 20A and the user terminal 20B are located at the same base B1, the voice control unit 152 synchronizes the voice output from the user terminal 20A with the voice output from the user terminal 20B. As described above, in this embodiment, the avatar A1 corresponding to the user U1 and the avatar A2 corresponding to the user U2 act together in the virtual space VS. Therefore, the voice output from the user terminal 20A (the voice heard by the avatar A1 in the virtual space VS) and the voice output from the user terminal 20B (the voice heard by the avatar A2 in the virtual space VS) are the same voice. In the example of FIG. 4A to FIG. 4C, for example, the avatar A3 utters "Thank you for your cooperation today" (see FIG. 4A and FIG. 4B) to the avatars A1 and A2 is output from both the user terminals 20A and 20B.

[0052] If the same voice is output at different times in the user terminal 20A and the user terminal 20B, it is difficult to hear, and users U1 and U2 feel uncomfortable. Therefore, the voice control unit 152 synchronizes the voice output in the user terminal 20A and the user terminal 20B. As a specific method of synchronization, for example, any of the following [Method A-1] to [Method A-3] may be adopted.

[0053] [Method A-1] A method for exchanging synchronization frames between user terminals 20 In method A-1, the audio control unit 152 transmits audio data to both user terminals 20A and 20B. When one user terminal 20 (user terminal 20A as an example) receives the audio data, it notifies the other user terminal 20 (user terminal 20B as an example) of the identification information of the audio data. The user terminal 20B, upon receiving the notification, notifies the user terminal 20A by using a synchronization frame that it is receiving audio data with the same identification information and the timing of sound playback based on the audio data. The user terminal 20A and the user terminal 20B play sounds at the same timing based on the synchronization frame. According to method A-1, each user terminal 20 receives audio data, so that the processing load related to synchronization can be reduced.

[0054] [Method A-2] A method in which one user terminal 20 acts as a master and provides voice data In method A-2, the audio control unit 152 transmits audio data to either the user terminal 20A or 20B (user terminal 20A as an example). The user terminal 20A transmits the audio data to the other user terminal 20 (user terminal 20B as an example) using short-range wireless. The user terminal 20A waits for a period of time that depends on the short-range wireless band, and then plays back the audio data. Furthermore, the user terminal 20B plays back the audio data immediately after receiving it. According to method A-2, it is possible to eliminate a discrepancy that depends on the difference in the communication environment between each user terminal 20 and the server 10.

[0055] [Method A-3] A method for setting a delay based on differences in communication methods between user terminals 20 In method A-3, the voice control unit 152 transmits voice data to both the user terminals 20A and 20B. The voice control unit 152 compares the communication method used by the user terminal 20A with the communication method used by the user terminal 20B. The voice control unit 152 delays the transmission timing of voice data to the user terminal 20 with the faster communication speed in accordance with the communication method of the user terminal 20 with the slower communication speed. For example, when the user terminal 20A supports 5G (5th Generation Mobile Communication System) but the user terminal 20B does not support 5G and uses 4G (4th Generation Mobile Communication System), the communication speed of the user terminal 20A is faster. Therefore, the voice control unit 152 delays the transmission timing of voice data to the user terminal 20A from the transmission timing of voice data to the user terminal 20B. The delay time is determined based on the difference in communication speed between 5G and 4G. Note that when the user terminals 20A and 20B use the same communication method, the timing of receiving voice data at the user terminals 20A and 20B is almost the same. Therefore, the audio control unit 152 simultaneously transmits audio data to the user terminals 20A and 20B. According to method A-3, each user terminal 20 only needs to output the received audio data as is, so that the processing load related to synchronization can be reduced.

[0056] [Audio input] When the user terminal 20A and the user terminal 20B are located at the same location B1, the voice control unit 152 controls the user terminal 20A to transmit voice data when the user U1 speaks, and controls the user terminal 20B to transmit voice data when the user U2 speaks. In other words, when the user terminal 20A and the user terminal 20B are located at the same location B1, the voice control unit 152 controls the user terminal 20A not to transmit voice data except when the user U1 speaks, and controls the user terminal 20B not to transmit voice data except when the user U2 speaks.

[0057] As described above, in this embodiment, two users U1 and U2 are located around the user terminals 20A and 20B. Therefore, if the microphones 204 of the user terminals 20A and 20B pick up the voice uttered by the user U1, and if all of the voice data are transmitted to the server 10, the sound may be difficult to hear. Therefore, the voice control unit 152 controls each user terminal 20 to transmit voice data only when the user U who owns each user terminal 20 speaks. In the example of FIG. 4A to FIG. 4C, for example, when the user U1 corresponding to the avatar A1 utters "Hello!", the user terminal 20A transmits the voice data of the collected voice to the server 10. Also, when the user U2 corresponding to the avatar A2 utters "It's been a while!", the user terminal 20B transmits the voice data of the collected voice to the server 10.

[0058] The voice data transmitted from the user terminals 20A and 20B to the server 10 is transmitted from the server 10 to the user terminal 20C. Since there are no other user terminals 20 at the location B2 where the user terminal 20C is located, the user terminal 20C outputs the received voice data as is (without performing synchronization processing).

[0059] In each user terminal 20, the method of recognizing what has been spoken by the user U of the own device may be, for example, any of the following [Method B-1] to [Method B-3].

[0060] [Method B-1] Predicting speech timing using user image In method B-1, each user terminal 20 analyzes a user image captured by the imaging device 206 to predict the timing of user U's speech. The processing device 209 of each user terminal 20 may function as a prediction unit. Predicting the timing of speech means, for example, predicting in advance that user U will start speaking. In advance may be, for example, several seconds to several hundred milliseconds before the start of speech. Each user terminal 20 analyzes the user image using a learning model that has learned the characteristics of the mouth movement of user U at the start of speech, and predicts that user U will start speaking. The mouth movement of user U may be rephrased as the arrangement of user U's facial features or the movement of user U's facial features.

[0061] The user terminal 20, which has predicted that the user U will start speaking, transmits voice data collected by the microphone 204 to the server 10. The timing to end the transmission of the voice data may be determined based on the movement of the mouth of the user U shown in the user image. Alternatively, the transmission of the voice data may be ended when the input level of the voice becomes equal to or lower than a predetermined threshold (first threshold). According to method B-1, the transmission of the voice data can be started before the user U actually starts speaking, so that the user U's speech can be conveyed to the other party without omission.

[0062] [Method B-2] Identify user U's voice quality and detect speech In method B-2, each user terminal 20 stores the voice quality of user U of the own device in advance and identifies whether the voice collected by microphone 204 is the voice of user U or not. The processing device 209 of each user terminal 20 may function as the identification unit. The user terminal 20 that detects that user U is speaking transmits voice data collected by microphone 204 to the server 10. The timing to end the transmission of the voice data may be, for example, when the input level of the voice collected by microphone 204 becomes equal to or lower than a predetermined threshold (first threshold), or when the voice of user U is no longer included in the voice collected by microphone 204. According to method B-2, the voice of user U can be detected with high accuracy with a relatively low processing load.

[0063] [Method B-3] Detecting speech based on input level to microphone 204 In method B-3, when the input level of the voice to the microphone 204 of the user terminal 20 becomes equal to or higher than a predetermined threshold (second threshold), the user terminal 20 determines that the user U of the user terminal 20 is speaking, and transmits the voice data collected by the microphone 204 to the server 10. In general, the user U who owns the user terminal 20 is the one who is closest to the user terminal 20. Therefore, for example, when the user U1 speaks, it is predicted that the input level of the microphone 204 of the user terminal 20A will be equal to or higher than the threshold. For example, the value of the current voice input level may be exchanged between the user terminals 20A and 20B, and the voice data collected by the user terminal 20 with the highest input level may be transmitted to the server 10. According to method B-3, it is possible to switch between the transmission and non-transmission of voice data by simple processing.

[0064] As described above, the voice data collected by the user terminals 20A and 20B are transmitted to the user terminal 20C via the server 10. Also, the voice data collected by the user terminal 20C is transmitted to at least one of the user terminals 20A and 20B via the server 10. On the other hand, the voice data collected by the user terminal 20A is not transmitted to the user terminal 20B located at the same location B1. Also, the voice data collected by the user terminal 20B is not transmitted to the user terminal 20A located at the same location B1. That is, when the user terminal 20A and the user terminal 20B are located at the location B1 and the user terminal 20C is located at the location B2, the voice control unit 152 transmits the first voice data collected by the user terminal 20A and the second voice data collected by the user terminal 20B to the user terminal 20C. Also, the voice control unit 152 transmits the third voice data collected by the user terminal 20C to at least one of the user terminals 20A and 20B. This is because, for example, the voice collected by the user terminal 20B located at the same location B1 as the user terminal 20A is heard directly by the user U1. In order to avoid overlapping of sounds, the voice control unit 152 does not transmit the voice data transmitted from one user terminal 20A located at the same location B1 to the other user terminal 20B.

[0065] [How to output audio based on avatar A's position] The sound control unit 152 may control the processing of sound data in the user terminal 20A and the user terminal 20B based on the positional relationship between the avatar A1 and the avatar A2 in the virtual space VS. Specifically, for example, when the avatar A1 is located closer to the sound source than the avatar A2 in the virtual space VS, the sound control unit 152 causes the sound generated from the sound source to be output louder from the user terminal 20A than from the user terminal 20B.

[0066] More specifically, consider a case where avatars A1 to A3 are positioned as shown in FIG. 5. When avatar A3 (user U3) is speaking, avatar A3 can be regarded as a sound source. Comparing the distance L1 between avatar A1 and avatar A3 and the distance L2 between avatar A2 and avatar A3, distance L1 is shorter. Therefore, comparing avatar A1 and avatar A2, it can be said that avatar A1 is closer to the sound source. In this case, the sound control unit 152 makes the output volume of the speech of avatar A3 in user terminal 20A larger than the output volume of the speech of avatar A3 in user terminal 20B.

[0067] Also, for example, when the speaker 205 of the user terminal 20A is a stereo speaker, the volume of the sound from the left and right speakers 205 may be changed based on the direction of the sound source. For example, consider a case where the sound source is on the right side as viewed from the avatar A1 and the sound source is on the left side as viewed from the avatar A2. For the user terminal 20A, the audio control unit 152 increases the output volume from the right speaker 205 more than the output volume from the left speaker 205. For the user terminal 20B, the audio control unit 152 increases the output volume from the left speaker 205 more than the output volume from the right speaker 205.

[0068] In this way, by changing the audio output mode (mainly the volume) based on the positional relationship between avatar A and the sound source in the virtual space VS, the experience in the virtual space VS can be made more realistic. Note that the sound source is not limited to avatar A3, but may be various objects placed in the virtual space VS or sounds based on the effects in the virtual space VS (wind sounds, etc.).

[0069] A-5. Operation of the processing device 103 Fig. 6 is a flowchart showing the operation of the processing device 103. The flowchart in Fig. 6 mainly shows the processing related to the user terminals 20A and 20B. The processing device 103 of the server 10 functions as a position acquisition unit 151 and acquires position information from each of the user terminals 20A and 20B (step S11). The processing device 103 functions as a voice control unit 152 and determines whether the position information of the user terminals 20A and 20B matches (step S12).

[0070] If the location information matches (step S12: YES), the processing device 103 functions as the voice control unit 152 and determines whether pairing (connection via short-range wireless communication) between the user terminals 20A and 20B is possible (step S13). If pairing is not possible (step S13: NO) or if the location information does not match in step S12 (step S12: NO), the processing device 103 determines that the user terminals 20A and 20B are not located at the same base B1, and excludes them from control (step S14).

[0071] On the other hand, if pairing is possible in step S13 (step S13: YES), the processing device 103 determines that the user terminals 20A and 20B are located at the same location B1, and controls the processing of voice data in the user terminals 20A and 20B. Specifically, if voice data is received from a user terminal 20C other than the user terminals 20A and 20B (step S15: YES), the processing device 103 functions as the voice control unit 152, transmits the received voice data to the user terminals 20A and 20B, and synchronizes the user terminals 20A and 20B (indicated as "user terminals" in the figure) to output sound based on the voice data (step S16). Note that if voice data is not received from the other user terminal 20C in step S15 (step S15: NO), the processing device 103 advances the processing to step S17.

[0072] Furthermore, when voice data is received from either the user terminal 20A or 20B (step S17: YES), the processing device 103 functions as the voice control unit 152 and transmits the received voice data to the user terminal 20C at another location B2 (step S18). The voice data transmitted to the user terminal 20C is output at the user terminal 20C. Furthermore, the voice data is not transmitted to the other user terminal 20 (e.g., the user terminal 20B) located at the same location B1 as the user terminal 20 (e.g., the user terminal 20A) that transmitted the voice data. Thereafter, the processing device 103 returns the process to step S11. Furthermore, when voice data is not received from the user terminal 20A or 20B at step S17 (step S17: NO), the processing device 103 also returns the process to step S11.

[0073] A-6. Summary of embodiments As described above, the server 10 according to the embodiment controls the processing of voice data in the user terminal 20A and the user terminal 20B based on whether the user terminal 20A and the user terminal 20B are located at the same location B1. This improves the convenience when participating in the same session using the user terminal 20A and the user terminal 20B.

[0074] Furthermore, when the user terminal 20A and the user terminal 20B are located at the same location B1, the server 10 synchronizes the audio output from the user terminal 20A with the audio output from the user terminal 20B. This reduces the time lag between the sounds output from the user terminal 20A and the user terminal 20B, making the output sounds in the session easier to hear.

[0075] Furthermore, when the user terminal 20A and the user terminal 20B are located at the same location B1, the server 10 controls the user terminal 20A to transmit voice data when the user U1 speaks, and controls the user terminal 20B to transmit voice data when the user U2 speaks. Thus, when one user U speaks, only the sound collected by one user terminal 20 is transmitted, making it easier for the other user terminal 20C to hear the sound.

[0076] In addition, when the user terminal 20A and the user terminal 20B are located at the location B1 and the user terminal 20C is located at the location B2, the server 10 transmits the voice data collected by the user terminal 20A and the second voice data collected by the user terminal 20B to the user terminal 20C, and transmits the voice data collected by the user terminal 20C to at least one of the user terminals 20A and 20B. In other words, the voice data collected by the user terminal 20B is not transmitted to the user terminal 20A, and the voice data collected by the user terminal 20A is not transmitted to the user terminal 20B. Therefore, for example, the speech voice of the user U2 located at the same location B1 as the user U1 is not transmitted to the user terminal 20A, so that the overlap of the voices is avoided and the output sound in the session is easy to hear.

[0077] Furthermore, the server 10 controls the processing of audio data in the user terminal 20A and the user terminal 20B based on the positional relationship between the avatar A1 and the avatar A2 in the virtual space VS. For example, when the avatar A1 is located closer to the sound source than the avatar A2 in the virtual space VS, the server 10 outputs the sound generated from the sound source louder from the user terminal 20A than from the user terminal 20B. Thus, the position of the avatar A in the virtual space VS can be reflected in the sound output from the user terminal 20, making the experience in the virtual space VS more realistic.

[0078] B: Variation The following are modified aspects of the above-described embodiment. Two or more aspects selected from the following modified aspects may be combined as appropriate within the scope of not being mutually contradictory.

[0079] B1: First modified example In the above-described embodiment, the session was conducted to facilitate communication between the avatars A1 to A3 (users U1 to U3) in the virtual space VS. The session is not limited to this, and may be, for example, a web conference connecting the user terminals 20A to 20C. The present embodiment is also applicable to cases where the audience (users U1 to U3) use the user terminal 20 to view, for example, a panel discussion by guests, video distribution by an individual, or a movie screening in the virtual space VS.

[0080] As shown in the first modified example, the system 1 according to the present embodiment is applicable to various situations.

[0081] B2: Second variant In the above-described embodiment, whether the user terminals 20A and 20B are located at the same base B1 is determined based on whether the user terminals 20A and 20B can be paired in addition to the location information. This is not limiting, and for example, when the location information matches, it may be determined that the user terminals 20A and 20B are located at the same base B1. For example, when the accuracy of the location information is about several meters, or when the relative position within a building can be specified using the location information, it may be determined whether the user terminals 20A and 20B are located at the same base B1 using only the location information. According to the second modification, it is possible to determine whether the user terminals 20A and 20B are located at the same base B1 by simple processing.

[0082] B3: Third variant In the above-described embodiment, the user U1 and the user U2 listen to the sound output from the speaker 205 of the user terminal 20A and the user terminal 20B, respectively. On the other hand, for example, the user U1 may listen to the sound using an earphone connected to the user terminal 20A. In such a state, the above-described control of the sound may not be appropriate. Therefore, the control of the sound described in the embodiment may be changed depending on whether the user U1 and the user U2 are wearing earphones, in other words, whether the earphones are connected to the user terminals 20A and 20B.

[0083] Hereinafter, a case where earphones are connected to the user terminal 20A will be described as an example. When earphones are connected to the user terminal 20A, the user terminal 20A notifies the server 10 that the earphones are connected to the user terminal 20A. The processing device 103 of the server 10 may function as a first terminal information acquisition unit that acquires information indicating whether or not earphones are used in the user terminals 20A and 20B. When earphones are connected to the user terminal 20A, the processing device 103 (voice control unit 152) of the server 10 transmits the voice of the user U2 to the user terminal 20A. As a result, even if the user U1 is wearing earphones and has difficulty directly hearing the voice of the user U2 at the same location B1, the user U2 can hear the voice of the user U2 transmitted from the server 10. For example, when the user U2 is not using earphones, the user U2 can directly hear the voice of the user U1. Therefore, the voice of the user U1 is not transmitted to the user terminal 20B as in the embodiment.

[0084] Furthermore, when user U1 is using earphones, the proportion of the sound output from speaker 205 of user terminal 20B that user U1 can hear is significantly smaller than when earphones are not worn. Furthermore, since no sound is output from speaker 205 of user terminal 20A, user U2 cannot hear the sound output from user terminal 20A. Therefore, there is almost no overlap between the sounds heard by user U1 and user U2.

[0085] Therefore, when earphones are connected to at least one of the user terminals 20A and 20B, the processing device 103 of the server 10 stops synchronization of audio output in the user terminals 20A and 20B. This reduces the processing load of the server 10, the user terminal 20A, and the user terminal 20B.

[0086] According to the third modification, when at least one of the user U1 and the user U2 is using earphones, the sound output is appropriately controlled.

[0087] B4: Fourth variant In the above-described embodiment, the input of sound is controlled so that there is no overlap in the sounds transmitted from the user terminals 20A and 20B. On the other hand, when the directivity of the microphones 204 of the user terminals 20A and 20B is high (when the microphones 204 are highly directional microphones), it is considered that there is almost no overlap in the sounds collected by each microphone 204. Therefore, when the directivity of at least one of the microphones 204 of the user terminals 20A and 20B is high, the processing device 103 (sound control unit 152) of the server 10 may disable the control of the sound input in the user terminal 20 having the microphone 204 with high directivity.

[0088] Whether the microphone 204 has high directivity or not is determined, for example, by determining in the user terminals 20A and 20B whether the model of the microphone 204 (including an external microphone) of the own device is a model that is registered in advance as a "highly directional microphone". If the model is registered as a highly directional microphone of the own device, the user terminal 20A or 20B notifies the server 10 of that fact. The registered name or Bluetooth (registered trademark) address of the microphone 204 may be used to identify the model of the "highly directional microphone". Alternatively, for example, the user terminals 20A and 20B may transmit model information of the microphone 204 of the own device to the server 10, and the server 10 may determine whether the model is a model that is registered in advance as a "highly directional microphone". Alternatively, for example, if the user terminal 20A has a highly directional microphone, the user U1 may manually register in the server 10 that the highly directional microphone is being used. The processing device 103 of the server 10 may function as a second terminal information acquisition unit that acquires information indicating whether or not the microphones 204 of the user terminals 20A and 20B are high-directional microphones.

[0089] According to the fourth modification, unnecessary processes are invalidated, so that the processing load on at least one of the server 10, the user terminal 20A, and the user terminal 20B is reduced.

[0090] C:Other (1) In the above-described embodiment, ROM and RAM, etc. are given as examples of storage devices 102 and 208, but storage devices 102 and 208 may also be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disc), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium.

[0091] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0092] (3) In the above-described embodiment, the input / output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or added. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.

[0093] (4) In the above-described embodiments, the determination may be made based on a value represented by 1 bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0094] (5) The order of the process steps, sequences, flow charts, etc. illustrated in the above-described embodiments may be changed without causing any inconsistency. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0095] (6) Each function illustrated in Figures 2 and 3 is realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by using wires, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.

[0096] (7) The programs exemplified in the above-described embodiments should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0097] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0098] (8) In each of the above embodiments, the terms “system” and “network” are used interchangeably.

[0099] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0100] (10) In the above-described embodiments, the portable device may be a mobile station (MS). A mobile station may be referred to by those skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other suitable term. In this disclosure, the terms "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.

[0101] (11) In the above-mentioned embodiments, the terms "connected" and "coupled" or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "accessed." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0102] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0103] (13) As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0104] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this disclosure is not intended to be an exclusive or.

[0105] (15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural.

[0106] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0107] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information). [Explanation of symbols]

[0108] 1...system, 10...server, 20 (20A to 20C)...user terminal, 120...avatar control unit, 130...virtual space control unit, 140...transmission data generation unit, 150...terminal control unit, 151...position acquisition unit, 152...voice control unit, 211...input control unit, 212...output control unit, A (A1 to A3)...avatar, B1, B2...locations, N...network, U (U1 to U3)...user, VD...virtual space data, VS...virtual space.

Claims

1. A session management device that manages sessions between a plurality of terminals including a first terminal and a second terminal, A location acquisition unit that acquires location information of the first terminal and location information of the second terminal; a voice control unit that controls processing of voice data in the first terminal and the second terminal based on whether the first terminal and the second terminal are located at the same base; A session management device comprising:

2. the audio control unit synchronizes an audio output from the first terminal with an audio output from the second terminal when the first terminal and the second terminal are located at the same base; 2. The session management device according to claim 1.

3. the first terminal is used by a first user, and the second terminal is used by a second user; When the first terminal and the second terminal are located at the same base, the voice control unit controls the first terminal to transmit voice data when the first user speaks, and controls the second terminal to transmit voice data when the second user speaks.

2. The session management device according to claim 1.

4. the plurality of terminals includes a third terminal; The voice control unit is When the first terminal and the second terminal are located at a first base and the third terminal is located at a second base, first voice data collected by the first terminal and second voice data collected by the second terminal are transmitted to the third terminal, and third voice data collected by the third terminal is transmitted to at least one of the first terminal and the second terminal.

2. The session management device according to claim 1.

5. the first terminal is used by a first user, and the second terminal is used by a second user; the session is transmission and reception of data for realizing a conversation in a virtual space between a plurality of avatars including a first avatar corresponding to the first user and a second avatar corresponding to the second user; the audio control unit controls processing of audio data in the first terminal and the second terminal based on a positional relationship between the first avatar and the second avatar in the virtual space; 2. The session management device according to claim 1.

6. the audio control unit, when the first avatar is located closer to a sound source than the second avatar in the virtual space, outputs a sound generated from the sound source louder from the first terminal than from the second terminal; 6. The session management device according to claim 5.

Citation Information

Patent Citations

  • On-line conference space construction system

    JP2023056245A