Voice distribution system, voice distribution method and program

Abstract

To provide a voice distribution system, a voice distribution method and a program that realize highly entertaining online Karaoke service.SOLUTION: In a voice distribution system for controlling distribution of voice data to a plurality of client devices 200 participating in a voice call, an API server 140 accepts singing requests from the plurality of client devices, a distribution server classifies the plurality of client devices according to acceptance for the singing requests, and, when a singing request is accepted, classifies the plurality of client devices into a first client device that performs singing and a second client device that does not perform singing according to the singing request, and distributes first voice data acquired from the first client device to the second client device, while not distributing second voice data acquired from the second client device to the first client device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an audio distribution system, an audio distribution method, and a program, and in particular to a technology for distributing binocular stereoscopic content. [Background technology]

[0002] There is an online karaoke server device that realizes online karaoke by utilizing an online conference system (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-083689 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in an online conference system, participants' voice data is transmitted and received via a server, so the contents of a participant's speech are inevitably transmitted to other participants with a delay from the timing at which the participant spoke. Therefore, if karaoke is performed in such a system environment, while one participant is singing, the voices of other participants reacting or singing along with the music are transmitted to the singer with a delay, which may impair the singing experience of the singer.

[0005] Therefore, in conventional online karaoke services, the call function is interrupted when any participant is singing, but this can reduce interest because participants are unable to communicate with each other every time they sing.

[0006] The present invention has been made in consideration of the above-mentioned problems, and has as its object to provide an audio distribution system, an audio distribution method, and a program for providing a highly entertaining online karaoke service. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the audio distribution system of the present invention is an audio distribution system that controls the distribution of audio data to multiple client devices participating in a voice call, and includes an audio acquisition means for acquiring audio data based on audio input in the multiple client devices, an audio distribution means for distributing audio data acquired from one client device to client devices among the multiple client devices excluding the one client device, a reception means for accepting singing requests from the multiple client devices, and a classification means for classifying the multiple client devices in accordance with the acceptance of a singing request, wherein when a singing request is accepted, the classification means classifies the multiple client devices into a first client device that will sing and a second client device that will not sing in accordance with the singing request, and the audio distribution means distributes the first audio data acquired from the first client device to the second client device while not distributing the second audio data acquired from the second client device to the first client device. Effect of the Invention

[0008] With this configuration, the present invention makes it possible to provide a highly entertaining online karaoke service. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an audio distribution system according to an embodiment and a modification of the present invention; [Diagram 2] FIG. 1 is a block diagram illustrating a hardware configuration of a distribution server 100 according to an embodiment and a modification of the present invention. [Diagram 3] FIG. 2 is a block diagram illustrating a hardware configuration of a client device 200 according to an embodiment and a modification of the present invention. [Figure 4] FIG. 1 is a diagram illustrating a karaoke room according to an embodiment and a modification of the present invention; [Diagram 5] FIG. 13 shows an example of a screen presented when using a karaoke function according to an embodiment and a modification of the present invention. [Figure 6] 1 is a flowchart illustrating an example of an audio distribution process executed by the distribution server 100 according to an embodiment and a modification of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Embodiment] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] In the embodiment described below, an example of an audio distribution system in which the present invention is applied to an audio distribution system capable of providing an online karaoke service in parallel with an audio call service in which multiple client devices participate is described. However, the present invention can be applied to any system capable of controlling the distribution of audio data related to audio calls acquired from multiple client devices and audio data related to singing inputted in any of the client devices to the client devices.

[0012] <Configuration of audio distribution system> The configuration of the audio distribution system of this embodiment will be described below with reference to FIG.

[0013] As shown in the figure, the devices constituting the audio distribution system of this embodiment are divided into a user side including client devices 200 used by users who use the service, and a provider side that provides services to these client devices 200. In Fig. 1, the devices on the provider side include a distribution server 100, an API (Application Programming Interface) server 140, a data server 150, and a voice chat (VC) server 160. These devices are configured to be communicatively connected via a network 300.

[0014] Here, the API server 140 serves as an interface for providing various functions of the service to the client device 200, and receives and processes requests for use of the various functions from the client device 200. A user can use a service provided by the audio distribution system, for example, by executing a dedicated service application on the client device 200. Thus, the API server 140 acquires a request sent based on an operation input performed by the user on the service application, and causes each device on the provider side to perform the necessary processing.

[0015] The service provided by the voice distribution system provides a voice call function to users. The voice call function is provided via a VC server 160. The VC server 160 acquires voice data (hereinafter referred to as call voice data) relating to a call based on voice input in a client device 200 via a service application. When the VC server 160 acquires the call voice data, it processes the voice data as necessary and sequentially distributes it to the client device 200 used by the user at the other end of the call. Through such operation of the VC server 160, users who are using the service and participating in a call can make a voice call with each other.

[0016] In addition to the voice call function, the service also provides a function for providing a karaoke experience (hereinafter referred to as the karaoke function). Although details will be described later, by using the karaoke function, for example, the client devices 200 used by multiple users invited by a host user are managed as a group, and voice calls and singing voices (karaoke voices) can be distributed by the client devices 200 in the group. Of the karaoke function, the VC server 160 is responsible for voice calls between users participating in the group, as in the case where the karaoke function is not used. Meanwhile, the distribution server 100 is responsible for distribution of karaoke voice data generated by participants. The distribution server 100 acquires karaoke voice data generated based on voice input in the client device 200 from the client device 200 used by the user who sings, and distributes it sequentially to the client devices 200 used by other users who are participating in the group. In the following description, among multiple users who use the karaoke function as one group, users who sing are clearly distinguished as "singers" and users who do not sing, i.e., users who watch the singer's singing, are clearly distinguished as "audiences." Singing is not limited to one user; a user using the karaoke function can be either a singer or an audience member depending on how they use the function.

[0017] In the audio distribution system of this embodiment, music data related to the karaoke function (including, for example, MIDI data (accompaniment sound source data) related to the accompaniment, melody line data, lyrics data, etc.) is supplied from the data server 150. The data server 150 stores music data for songs that can be sung in the karaoke function of the service. When the API server 140 acquires information specifying the song to be sung from the singer's client device 200, it returns information for accessing the corresponding music data stored in the data server 150 (hereinafter referred to as music information) to the client device 200. This allows the service application executed on the singer's client device 200 to download the music data from the data server 150. When the music data is downloaded, the service application plays the accompaniment sound source, and the singer can sing along with it to perform karaoke. As described above, the service application generates karaoke voice data based on the voice input of the singer's singing and sends it to the distribution server 100, and the voice data is distributed to the audience's client device 200 via the distribution server 100.

[0018] In addition, in order to increase the interest of the karaoke experience, the service provided by the audio distribution system of this embodiment provides at least the karaoke function by providing a virtual space corresponding to a karaoke room for a user group, and enables visual communication between users in the virtual space. An avatar corresponding to each user is placed in the virtual space, and the virtual space is rendered and displayed on the client device 200, so that the user can visually appreciate the state of karaoke being performed in the karaoke room together with other users. In this embodiment, the client device 200 is a device capable of viewing binocular stereoscopic content such as an HMD, and the service application renders the state of the virtual space from each of the left eye viewpoint and the right eye viewpoint, thereby providing an experience of viewing as if the user were present in the karaoke room.

[0019] The state (behavior) of an avatar arranged as an alter ego of each user in a virtual space is controlled based on an input made on the client device 200 used by the user. The state control of the avatar may be performed based on an operation input related to a movement or a change in a facial expression made by the user, an operation input related to an emote selection, or the like, or may be performed based on a voice uttered by the user or a detection result of a change in the position / posture of the HMD or the controller. The state data used for such avatar state control is acquired from each client device 200 via the API server 140, and is shared with other client devices 200 by the distribution server 100. In each client device 200, the service application changes the state of each avatar arranged in the virtual space based on the state data of each avatar shared via the distribution server 100, and presents the user with a binocular stereoscopic image related to the karaoke function by rendering the virtual space based on the viewpoint of the avatar corresponding to the client device 200. Here, the viewpoint of the avatar can be set based on, for example, the position of the avatar's head and the direction in which its front faces. The distribution server 100 is also used to share with the group's client devices 200 information about the song being sung, information indicating the state of objects placed in the karaoke room, information about the effects that will be generated, and the like.

[0020] As described above, the service provided by the audio distribution system of this embodiment realizes communication between users via a virtual space, and therefore, hereinafter, this service is referred to as a "metaverse service."

[0021] Configuration of distribution server 100 Here, the hardware configuration of the distribution server 100 in the distribution system of this embodiment will be described with reference to the block diagram of FIG.

[0022] The control unit 101 is, for example, a CPU, and controls the operation of each block of the distribution server 100. Specifically, the control unit 101 reads out an operation program for each block recorded in, for example, a storage device 102, expands the program in the memory 103, and executes the program to control the operation of each block.

[0023] The storage device 102 is a recording device capable of permanently retaining data, such as a non-volatile memory or a HDD. The storage device 102 stores information such as parameters required for the operation of each block in addition to the operation programs of each block of the distribution server 100. The storage device 102 also serves as a database (DB) for managing information required for providing the karaoke function. In the distribution server 100 of this embodiment, the storage device 102 includes a master DB 110 for managing music data of songs that can be sung in the karaoke function and basic information for displaying a karaoke room, such as the external configuration of the karaoke room. The storage device 102 also includes a user DB 120 for managing information about users who use the service application. The user DB 120 is used, for example, to manage information for identifying the external appearance of an avatar corresponding to each user and to manage gifts given to a singer by an audience member when the karaoke function is used. The storage device 102 also includes a room DB 130 for managing karaoke rooms currently opened.

[0024] In this embodiment, the storage device 102 of the distribution server 100 is described as functioning as various DBs, but the embodiment of the present invention is not limited to this. It goes without saying that at least one of the master DB 110, the user DB 120, and the room DB 130 may use an external database server different from the distribution server 100.

[0025] The memory 103 is a storage device such as a volatile memory used for temporary data storage. The memory 103 is used not only as an area for developing the operation programs of each block, but also as a storage area for temporarily storing data output during the operation of each block. The memory 103 also functions as a cache for temporarily storing information to be shared / distributed to the client devices 200 of the group when the karaoke function is being used.

[0026] The communication I / F 104 is a communication interface with an external device that the distribution server 100 has. The communication I / F 104 can connect to the external device via a network 300 (whether wired or wireless) and enable data transmission and reception. The communication I / F 104 converts, for example, information input as a transmission target, into data of a predetermined format and transmits the data to the external device via the network 300. Furthermore, when the communication I / F 104 receives information from an external device via the network 300, it decodes the information as necessary and stores the information in the memory 103.

[0027] Configuration of the Client Device 200 Next, the hardware configuration of the client device 200 of this embodiment will be described with reference to the block diagram of FIG.

[0028] The control unit 201 is, for example, a CPU, and controls the operation of each block included in the client device 200. Specifically, the control unit 201 reads out, for example, an operation program for each block or a service application program recorded in the storage device 202, and loads and executes the program in the memory 203 to control the operation of each block.

[0029] The storage device 202 is a recording device capable of permanently retaining data, such as a non-volatile memory or a HDD. The storage device 202 stores information such as parameters required for the operation of each block, in addition to the operation programs of each block and the service application programs of the client device 200. The memory 203 is a storage device used for temporary data storage, such as a volatile memory. The memory 203 is used not only as an area for expanding the operation programs of each block, but also as a storage area for temporarily storing data output in the operation of each block.

[0030] The rendering device 204 is, for example, a rendering device such as a GPU or a graphic chip. The rendering device 204 generates a screen that enables binocular stereoscopic vision of a virtual space while the metaverse service is being used. While the karaoke function is being used, the rendering device 204 executes a rendering process based on various information (avatar status, karaoke room status, etc.) received regarding the karaoke room in which the user is participating. More specifically, the rendering device 204 arranges the avatars of the participating users in a three-dimensional space corresponding to the karaoke room in a manner based on the status data. The rendering device 204 then generates screens for the left eye and the right eye by drawing the three-dimensional space in which the avatars are arranged based on the viewpoint of the avatar related to the user of the client device 200. By changing the parameters of the viewpoint according to the movement of the user's head, it is possible to provide a viewing experience as if the user were at a live concert venue. Here, the movement of the user's head may be identified based on the sensor output of a posture sensor 231 built into the HMD, which is the display device 230, for example. The attitude sensor 231 may be, for example, an acceleration sensor or a gyro sensor, and a sensor output of the attitude sensor 231 is acquired via an input I / F 205 described later. A screen generated by the drawing device 204 is displayed on the display device 230 via an output I / F 206.

[0031] The input I / F 205 is an interface that accepts various inputs provided to the client device 200. In the client device 200 of this embodiment, the input I / F 205 not only acquires a sensor output from the attitude sensor 231 provided in the display device 230, but also acquires information regarding an operation input made at the input device 210 such as a controller or a keyboard. The input I / F 205 stores information regarding the accepted input in, for example, the memory 203. In addition, the input I / F 205 also acquires an audio signal input via the microphone 220. The input I / F 205 converts the information regarding the accepted input into data of a predetermined format by performing A / D conversion processing or the like as necessary, and stores the converted information in, for example, the memory 203.

[0032] The output I / F 206 is an interface that outputs various information for presenting information provided in the client device 200. In the client device 200 of the present embodiment, in order to provide a karaoke experience to the user, a screen showing a three-dimensional space related to the karaoke room is displayed on the display device 230, and audio data related to the karaoke experience is output from the speaker 232 via the output I / F 206.

[0033] The communication I / F 207 is a communication interface with an external device that the client device 200 has. The communication I / F 207 can connect to an external device via a network 300 (whether wired or wireless) to enable transmission and reception of data. For example, the communication I / F 207 converts information input as a transmission target into data of a predetermined format, and transmits the data to an external device such as the API server 140 via the network 300. When the communication I / F 207 receives information from an external device via the network 300, it decodes the information as necessary and stores the information in the memory 203.

[0034] Karaoke function overview The karaoke function provided in the audio distribution system of this embodiment will be outlined below.

[0035] When an operational input related to the use of the karaoke function is made in the client device 200 while the service application is running, the provision of the service related to the karaoke function to the user is started. The operational input related to the use of the karaoke function may be received, for example, by selecting a menu related to the karaoke provided for the metaverse service, or may be received by moving to a predetermined facility in a virtual space provided for the metaverse service.

[0036] A user who will be the host (hereinafter simply referred to as the host) performs a procedure for opening a new karaoke room in the service application. Through this procedure, a room opening request is sent from the host's client device 200 to the API server 140, and the API server 140 performs processing for opening the karaoke room. The processing for opening the karaoke room includes, for example, issuing identification information (room ID) that uniquely identifies the newly opened karaoke room, causing the VC server 160 to generate a voice call channel for the karaoke room, and registering the room ID, the host's user ID, and information on the voice call channel in the room DB 130 as information on the new karaoke room (room information).

[0037] When a new karaoke room is opened, the host client device 200 acquires various information used to display the room from the distribution server 100, starts displaying a screen showing the interior of the room, and enters a state in which the user has entered the room. The host client device 200 also acquires connection destination information of the voice call channel of the VC server 160 corresponding to the karaoke room from the distribution server 100, and connects to the voice call channel to become capable of transmitting and receiving call voice data. The host client device 200 is also provided with a room ID of the opened karaoke room. The host can invite other users to the karaoke room as a guest by communicating the room ID to other users via an SNS or a message application, for example.

[0038] A user who will become a guest (hereinafter simply referred to as a guest) performs a procedure for entering an existing karaoke room in the service application. The karaoke room to be entered can be specified, for example, by inputting a room ID provided by the host. In addition, in a mode in which the karaoke room is open to the public and any user can enter the room, the room may be specified based on an operation input for selecting a room from a list of karaoke rooms that can be entered. Through this procedure, an entry request is transmitted from the guest's client device 200 to the API server 140, and the API server 140 performs a process for entering the corresponding karaoke room. The process for entering the karaoke room includes, for example, adding the guest's user ID to the corresponding room information managed in the room DB 130.

[0039] When the process for entering the karaoke room is completed, the guest's client device 200 acquires various information used to display the room from the distribution server 100, starts displaying a screen showing the interior of the room, and enters a state of having entered the room. The guest's client device 200 also acquires information on the connection destination of the voice call channel of the VC server 160 corresponding to the karaoke room from the distribution server 100, and becomes capable of connecting to the voice call channel and transmitting / receiving call voice data.

[0040] Once in the karaoke room (participating state), the host and guest can use various menus related to karaoke while making a voice call via the VC server 160. Each user can select a song they want to sing from the song list and perform a request operation. When a request operation is performed, a singing request is sent from the client device 200 to the API server 140. When a singing request is made, the API server 140 determines a user who will be the singer in the distribution server 100. When singing requests from multiple client devices 200 compete, the control unit 101 determines the user of one of the client devices 200 as the singer based on a predetermined rule. When the singer is determined, the other users participating in the karaoke room become the audience. That is, in response to receiving a singing request, the control unit 101 classifies the client devices 200 used by the participating users into the client devices 200 of the singer and the client devices 200 of the audience, and controls the audio distribution during karaoke performance. In this embodiment, the period from when a user is selected as the singer to when the singer completes singing the song (karaoke), i.e., when the reception of karaoke audio data from the singer's client device 200 is completed (when information relating to the end of singing is received), is referred to as the singer's "karaoke in progress."

[0041] As described above, when karaoke is performed online, the voice of the audience interferes with the singer's favorable singing experience. For this reason, in the audio distribution system of this embodiment, the control unit 101 performs control so that the singer's client device 200 does not receive voice data from the voice call channel corresponding to the karaoke room while karaoke is being performed. This control can be realized, for example, by the service application performing a process to leave the voice call channel in response to a command to temporarily leave the voice call channel being sent from the distribution server 100 to the singer's client device 200. Alternatively, this control may be realized by operating the VC server 160 so as not to transmit or receive voice data to or from the singer's client device 200.

[0042] Furthermore, the control unit 101 accumulates in a cache information (necessary information) required to start karaoke of the song to be sung (hereinafter referred to as the target song), and distributes it to each client device 200. The necessary information includes, for example, information indicating the user who has become the singer, information indicating the target song, and information on a path to the song data of the target song. The singer's client device 200 downloads the song data from the data server 150 based on the necessary information. That is, by transmitting the necessary information from the distribution server 100 to the singer's client device 200, distribution of song data from the provider side to the user side is realized within the audio distribution system.

[0043] The service application then provides the singer with a karaoke experience based on the downloaded music data. More specifically, the service application records the input voice as vocal voice data while playing backing sound source data of the target music and displaying lyrics based on the received music data. The recorded vocal voice data is sequentially stored in the memory 203, and the vocal voice data and the backing sound source data are synthesized to generate karaoke voice data, which is then sequentially sent to the distribution server 100.

[0044] Here, a time lag occurs between the playback of the accompaniment sound source data in the singer's client device 200 and the voice signal input by the singer in response to the playback being stored as vocal voice data. Therefore, when generating karaoke voice data, a process is performed to adjust the timing of the accompaniment sound source data and the vocal voice data and synthesize them. For example, the service application delays the playback timing of the accompaniment sound source data by the time lag, and synthesizes it with the vocal voice data generated at that time to generate karaoke voice data with adjusted timing.

[0045] The degree of the time lag may vary depending on the performance of the client device 200, the operating system used in the client device 200, the voice input device connected to the client device 200, etc. Therefore, the service application identifies the degree of the time lag and executes the synthesis process. Simply put, the service application can obtain information on the time lag (delay time) that is predetermined for each operating system used in the client device 200 and execute the synthesis process.

[0046] In this way, karaoke voice data in a suitable state is transmitted from the singer's client device 200 to the distribution server 100. When the distribution server 100 acquires karaoke voice data from the singer's client device 200, it distributes the voice data to the audience's client device 200. That is, the distribution server 100 controls so that the call voice data acquired from the audience's client device 200 is not distributed to the singer's client device 200 while the karaoke is being performed, and controls so that the karaoke voice data acquired from the singer's client device 200 is distributed to the audience's client device 200. Since the audience's client device 200 is connected to the voice call channel even during the karaoke performance, the karaoke voice data and the call voice data are distributed to the client device 200 from the provider to the user in parallel during the karaoke performance. That is, the audience can make voice calls with other audience members while listening to the voice of the singer singing the target song during the karaoke performance.

[0047] Incidentally, from the viewpoint of not impairing the singer's singing experience, it is sufficient to simply control not to distribute the call voice data related to the audience to the singer's client device 200. In other words, the singer's singing experience is not impaired even if the method of distributing karaoke voice data to the audience's client device 200 using the voice call channel. On the other hand, the voice call realized by the VC server 160 is a service that requires real-time performance, so the bit rate of the call voice data received from each client device 200 is set to be low. In addition, there is a possibility that the karaoke voice data is not provided to the audience with a sound quality that is suitable for listening in the voice call, for example, the voice is interrupted by the synthesis of the call voice data of other users. Therefore, in the voice distribution system of this embodiment, the service application generates karaoke voice data at a bit rate higher than the call voice data, and distributes the karaoke voice data independently via the distribution server 100, rather than synthesizing it with other call voice data via the voice call channel. This allows the audience to enjoy the singer's singing in a suitable manner.

[0048] When the singer finishes singing the target song, the control unit 101 cancels the classification of the participating users in the karaoke room. That is, the control unit 101 ends the control of differentiating the voice distribution between the singer's client device 200 and the audience's client device 200 during the karaoke performance, and cancels the classification of the client device 200. The control unit 101 also controls the client device 200 of the user who was the singer to receive the call voice data from the voice call channel corresponding to the karaoke room again. This control can be realized, for example, by the service application performing a process of reconnecting to the voice call channel in response to a reconnection command to the voice call channel being sent from the distribution server 100 to the client device 200 of the user who was the singer. This allows the user who was the singer to return to the voice call again, and to resume conversation with the participating users as before the karaoke performance.

[0049] Display Control In order to enhance the immersive feeling of karaoke, the karaoke room is provided with a structure such as that shown in Fig. 4. In the example of Fig. 4, the karaoke room is divided into a stage area 401 and an observation area 402, each of which has a different role. The stage area 401 and the observation area 402 are adjacent to each other, and for example, the floor surface of the stage area 401 is provided at a higher position than the observation area 402.

[0050] During the period when karaoke is not being performed, the avatars of all participating users are placed in the viewing area 402. That is, all participating users can enjoy voice calls with other users in the viewing area 402. Meanwhile, during karaoke performance, the avatars 404 of the audience users are kept placed in the viewing area 402, but the avatars 403 of the singer users are changed to the stage area 401. The singer users are placed in the stage area 401 with their avatars 403 facing the viewing area 402, so that the avatars 404 of the audience users in the viewing area 402 can be seen from the stage. That is, during karaoke performance, the singer avatar 403 and the audience avatars 404 face each other across the boundary between the stage area and the viewing area. In this way, the singer who is performing karaoke can be provided with a singing experience on stage where he or she is being noticed by the audience. Audience members can also enjoy the experience of watching a singer's performance on stage while having voice calls with other members of the audience.

[0051] In addition, display device objects (hereinafter referred to as monitor objects) 405 such as a screen or monitor are arranged in the stage area 401, and various information such as performance displays and lyrics are displayed while karaoke is being performed. That is, the contents of the texture applied to the information display surface of the monitor object 405 can be dynamically changed based on, for example, the necessary information received or information sequentially supplied from the distribution server 100. In this embodiment, the lyrics of the target song are displayed on the monitor object 405 while karaoke is being performed.

[0052] The lyrics display is not displayed so that the singer can check the lyrics while singing, but is mainly used to inform the audience of the lyrics of the target song. Therefore, the monitor object 405 is placed in a position in the stage area 401 farther away than the singer's avatar 403 when viewed from the viewing area 402, as shown in Fig. 4. In other words, by adjusting the viewpoint so as to capture the monitor object 405, the audience can know the lyrics of the target song through the monitor object 405, as shown in Fig. 5(a).

[0053] On the other hand, if the singer were to sing karaoke while checking the lyrics displayed on the monitor object 405, the singer's avatar 403 would turn his back to the audience's avatars 404 in the stage area 401, which would not provide a particularly pleasant viewing experience for the audience. For this reason, the lyrics of the target song are presented to the singer in a different manner. The lyrics are presented to the singer by placing a lyrics object 406 corresponding to the lyrics in the viewing area 402. The lyrics object 406 can be configured as, for example, a flat object to which a character string image showing the lyrics is applied. In other words, the character string shown by the lyrics object 406 can be dynamically changed based on, for example, the lyrics information received together with the music data.

[0054] Since the lyrics object 406 is displayed so that the singer can check the lyrics while singing, the lyrics object 406 may be controlled to be placed in three-dimensional space only on the singer's client device 200. Therefore, the lyrics object 406 is placed in a position farther away than the audience avatars 404 when viewed from the stage area 401 in the viewing area 402 as shown in Fig. 4. In other words, by adjusting the viewpoint to capture the lyrics object 406, the singer can control the behavior of the avatar 403 so that the singer naturally performs a singing performance facing the audience while checking the lyrics of the target song through the lyrics object 406 as shown in Fig. 5(b).

[0055] Audio distribution processing The audio distribution process performed in the audio distribution system of this embodiment will be described below in detail with reference to the flowchart in FIG. 6. The process corresponding to the flowchart can be realized by the control unit 101 of the distribution server 100 reading out the corresponding processing program stored in the storage device 102, for example, and developing and executing the program in the memory 103. The audio distribution process is described as being executed when the API server 140 receives a singing request from any of the client devices 200 participating in a specific karaoke room (hereinafter referred to as a target room). Although not described in detail, the control unit 101 controls the VC server 160 to sequentially acquire call voice data from the client devices 200 participating in the target room during the period when the target room is open, not limited to the audio distribution process, and to sequentially distribute the call voice data to the client devices 200 different from the sender. When distributing the call voice data, the VC server 160 may combine multiple call voice data as necessary.

[0056] In S601, the control unit 101 determines a singer based on a singing request. The control unit 101 also classifies the client devices 200 participating in the target room into client devices 200 of singers and client devices 200 of audience members based on the singing request.

[0057] In S602, the control unit 101 controls the singer's client device 200 to a state in which it does not receive call voice data from the voice call channel related to the target room. The process of this step is performed by transmitting a temporary leave command from the voice call channel to the singer's client device 200 via the communication I / F 104. In response to the temporary leave command, the service application running on the singer's client device 200 performs a process of leaving the connected voice call channel.

[0058] In S603, the control unit 101 accumulates necessary information for the target song related to the singing request in the memory 103, and then distributes the necessary information to the client devices 200 participating in the target room. Upon receiving the necessary information, the service application running on the singer's client device 200 starts downloading the song data, and then, when it becomes possible to start karaoke of the target song, sends information to that effect to the room DB 130.

[0059] In S604, the control unit 101 judges whether the singer's client device 200 is ready to start karaoke. If the control unit 101 judges that the singer's client device 200 is ready to start karaoke, the process proceeds to S605, and if the control unit 101 judges that the singer's client device 200 is not ready to start karaoke, the process of this step is repeated.

[0060] In S605, the control unit 101 starts acquiring karaoke voice data from the singer's client device 200. After acquiring the karaoke voice data, the control unit 101 distributes the karaoke voice data to the audience's client devices 200 via the communication I / F 104. As a result, the karaoke voice data from the distribution server 100 and the call voice data from the VC server 160 are distributed in parallel to the audience's client devices 200. At this time, the service application also performs display control related to the karaoke being performed on each of the client devices 200 participating in the target room. That is, the singer's avatar is placed in the stage area, the texture of the lyrics of the target song is applied to the monitor object 405, and a three-dimensional space in which the lyrics object related to the target song is placed in the appreciation area is presented to each user as a karaoke room.

[0061] In S606, the control unit 101 judges whether the singer has finished singing the target song. The judgment in this step may be made based on whether information regarding the end of singing has been received from the singer's client device 200. If the control unit 101 judges that the singer has finished singing the target song, it moves the process to S607, and if it judges that the singer has not finished singing the target song, it repeats the process in this step.

[0062] In S607, the control unit 101 controls the singer's client device 200 to a state in which it receives call voice data from the voice call channel related to the target room. The process of this step is performed by transmitting a reconnection command to the voice call channel to the singer's client device 200 via the communication I / F 104. In response to the reconnection command, the service application running on the singer's client device 200 performs a process of reconnecting to the voice call channel related to the target room.

[0063] In S608, the control unit 101 removes the classification of the client device 200 participating in the target room based on the singing request, and ends this voice distribution process.

[0064] As described above, the audio distribution system of this embodiment can provide a highly entertaining online karaoke service. More specifically, the audio distribution system provides a karaoke experience by isolating the singer's client device 200 from the voice call, while simultaneously realizing voice calls between the audience's client devices 200 while distributing karaoke voice data related to the singer's singing to the audience's client devices 200. In other words, the singer can enjoy the karaoke experience without being interrupted while singing, and the audience can watch the singer's singing while making voice calls with other audience members.

[0065] In the audio distribution system of this embodiment, the distribution server 100, API server 140, data server 150, and VC server 160 are configured to perform various functions as the configuration of the provider side, but the implementation of the present invention is not limited to this. All of the various functions of the provider side may be provided by one device, or may be provided separately by any number of devices.

[0066] In addition, in the audio distribution system of this embodiment, the singer's client device 200 has been described as synthesizing vocal voice data and accompaniment sound source data to generate karaoke voice data and sending it to the distribution server 100, but the embodiment of the present invention is not limited to this. For example, the singer's client device 200 may send vocal voice data with synchronization information such as a time stamp for synchronizing the vocal voice data with the accompaniment sound source data, and the distribution server 100 may generate karaoke voice data based on the synchronization information and distribute it to the audience's client devices 200.

[0067] [Variations] In the above embodiment, the present invention is applied to a metaverse service that provides a voice call function and a karaoke function with the display of a plurality of avatars arranged in a virtual space, but the implementation of the present invention is not limited to this. The present invention does not need to be performed with a display related to the virtual space, and can also be applied to an embodiment in which some of the users of the client devices 200 participating in a platform that simply provides communication only with voice are singers and the users of the other client devices 200 are audiences. In other words, the implementation of the present invention does not necessarily require the display related to the virtual space.

[0068] [Other embodiments] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0069] 100: distribution server, 101: control unit, 102: storage device, 103: memory, 104: communication I / F, 110: master DB, 120: user DB, 130: room DB, 140: API server, 150: data server, 160: VC server, 200: client device, 201: control unit, 202: storage device, 203: memory, 204: drawing device, 205: input I / F, 206: output I / F, 207: communication I / F, 210: input device, 220: microphone, 230: display device, 231: attitude sensor, 232: speaker, 300: network

Claims

1. 1. A voice distribution system for controlling distribution of voice data to a plurality of client devices participating in a voice call, comprising: a voice acquisition means for acquiring voice data based on a voice input in the plurality of client devices; a voice distribution means for distributing voice data acquired from one client device to other client devices other than the one client device among the plurality of client devices; A reception means for receiving singing requests from the plurality of client devices; a classification means for classifying the plurality of client devices in response to the acceptance of the singing request; having When the singing request is accepted, the classification means classifies the plurality of client devices into a first client device that is caused to sing and a second client device that is not caused to sing in response to the singing request; The audio distribution means distributes first audio data acquired from the first client device to the second client device, while not distributing second audio data acquired from the second client device to the first client device. An audio distribution system comprising:

2. 2. The audio distribution system according to claim 1, wherein the audio distribution means distributes the first audio data and the second audio data in parallel to the second client device when the singing request is received.

3. The audio distribution system according to claim 1, characterized in that, when the audio distribution means receives the singing request, it distributes the first audio data and the second audio data to the second client device without synthesizing them.

4. 2. The audio distribution system according to claim 1, wherein the audio distribution means distributes the first audio data at a bit rate higher than that of the second audio data.

5. 2. The audio distribution system according to claim 1, wherein the classification means cancels classification of the plurality of client devices when singing in the first client device ends.

6. Each of the plurality of client devices A generating means for generating voice data based on a voice input; a voice transmission means for transmitting the voice data generated by the generation means; an audio playback means for receiving and playing back the audio data distributed by the audio distribution means; having The audio reproducing means of the second client device reproduces the received first audio data and the received second audio data in parallel.

6. The audio distribution system according to claim 1,

7. the voice call is performed accompanied by display of an avatar associated with each of the plurality of client devices arranged in a virtual space; The audio distribution system includes: a state acquisition means for acquiring, from each of the plurality of client devices, state data relating to a state of an avatar associated with the client device; a sharing means for sharing the status data acquired by the status acquisition means with the plurality of client devices; and Each of the plurality of client devices a status transmission means for transmitting status data relating to an avatar associated with the client device; a display control means for causing a display means to display the virtual space in which avatars associated with each of the plurality of client devices are arranged based on the state data shared by the sharing means; Further having 7. The audio distribution system according to claim 6.

8. The audio distribution system according to claim 7, characterized in that the display control means of the first client device causes the display means to display the virtual space in which lyrics objects indicating lyrics corresponding to the singing request are further arranged.

9. the virtual space includes a first area in which an avatar associated with each of the plurality of client devices is placed when the request to sing has not been accepted, and a second area adjacent to the first area in which an avatar associated with the first client device is placed when the request to sing has been accepted, The lyrics object is disposed in the first area.

9. The audio distribution system according to claim 8.

10. 10. The audio distribution system according to claim 9, wherein the lyrics object is located at a position farther away from the second area than an avatar associated with the second client device.

11. A display object for displaying information is further disposed in the second area; 10. The audio distribution system according to claim 9, wherein the display control means, when receiving the singing request, causes the display means to display lyrics corresponding to the singing request via the display object.

12. 7. The audio distribution system according to claim 6, wherein the audio distribution means, when receiving the singing request, distributes music piece data corresponding to the singing request to the first client device.

13. In the first client device, The audio playback means plays back the music data, The generating means generates the second voice data based on a voice input during at least the reproduction of the music data.

13. The audio distribution system according to claim 12.

14. The audio distribution system described in claim 13, characterized in that the generation means of the first client device generates the second audio data by synthesizing audio data based on audio input during playback of the music data with the music data.

15. 1. A method for controlling delivery of voice data to a plurality of client devices participating in a voice call, comprising: a voice acquisition step of acquiring voice data based on voice input in the plurality of client devices; a voice distribution step of distributing voice data acquired from one client device to other client devices among the plurality of client devices other than the one client device; a receiving step of receiving singing requests from the plurality of client devices; a classification step of classifying the plurality of client devices in response to the acceptance of the singing request; having When the singing request is accepted, In the classifying step, the plurality of client devices are classified into a first client device that is caused to sing and a second client device that is not caused to sing in response to the singing request; In the audio distribution step, first audio data acquired from the first client device is distributed to the second client device, while second audio data acquired from the second client device is not distributed to the first client device.

13. An audio distribution method comprising:

16. A program for controlling delivery of voice data to a plurality of client devices participating in a voice call, comprising: On the computer, a voice acquisition process for acquiring voice data based on voice input in the plurality of client devices; an audio distribution process for distributing audio data acquired from one client device to the client devices other than the one client device among the plurality of client devices; a reception process for receiving singing requests from the plurality of client devices; a classification process for classifying the plurality of client devices in response to the acceptance of the singing request; Run the command, When the singing request is accepted, In the classification process, the plurality of client devices are classified into a first client device that is caused to sing and a second client device that is not caused to sing in response to the singing request; In the audio distribution process, first audio data acquired from the first client device is distributed to the second client device, while second audio data acquired from the second client device is not distributed to the first client device. A program characterized by:

Images