Apparatus and method for transmitting spatial audio using multicast

Abstract

To efficiently transmit spatial audio data of multi-player contents using multicast.SOLUTION: A spatial audio transmission device using multicast includes multiple servers that are connected to communicate to multiple client terminals executing multiplayer real-time contents and are separated by multicast groups. A server included in the multicast group, upon receiving first player information from a client terminal executing the multiplayer real-time content, multicast-transmits the first player information to the other servers included in the multicast group and so as to be shared, and upon receiving second player information from the other servers included in the multicast group, mixes the spatial audio based on the first player information and the second player information and unicast-transmits to the client terminals.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to a spatial audio transmission device, and more particularly to a spatial audio transmission device and method using multicast that can efficiently transmit spatial audio data of multi-player content using multicast. [Background technology]

[0002] Generally, in multiplayer real-time content such as online games, each player's data, such as the player's position and voice chat data, needs to be shared with other players.

[0003] When N players transmit data to the remaining N-1 players, a total of N(N-1) pieces of data must be transmitted, so the more data that is shared, the greater the network load will be, exponentially.

[0004] In the case of audio data, each player needs to collect all the multiple channel audio data received from other players and convert it into audio data supported by the player device (typically 2 channel stereo).

[0005] However, audio data conversion (audio mixing) has a problem in that the processor load increases according to the number of audio input channels.

[0006] In particular, in the case of mobile devices, the processor performance is inferior to that of general PCs, so as the processor load increases, the battery consumption also increases, and therefore it is necessary to limit the number of input channels.

[0007] To solve this problem, in the case of voice chat, instead of sending audio data directly from one player device to another, the audio data is sent to an audio streaming server, and the audio received from each player device is mixed in the audio streaming server and sent to each player device, thereby solving the problems of network traffic and processor load caused by audio mixing.

[0008] However, in the case of spatial audio, the audio generated varies for each player depending on the distance and direction from the sound source, so the audio must be mixed differently for each player.

[0009] As such, mixing the audio differently requires more computational complexity, which can lead to problems in that it is difficult for one audio streaming server to mix the audio of all players participating in the content, and this still requires N(N-1) data transmissions.

[0010] Therefore, there is a demand for the development of a spatial audio transmission device that can efficiently transmit spatial audio data in multi-player content. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] Korean Patent No. 10-2331229 Summary of the Invention [Problem to be solved by the invention]

[0012] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a spatial audio transmission device and method using multicast, which can efficiently transmit spatial audio data by multicasting and sharing player information of multi-player real-time content received from client terminals to servers included in a multicast group, and mixing spatial audio data based on the shared player information, thereby eliminating the transmission of duplicated data, reducing the load on the network, and transmitting spatial audio data efficiently.

[0013] The problems to be solved by the present disclosure are not limited to those mentioned above, and other problems not mentioned will be clearly understood by a person skilled in the art from the following description. [Means for solving the problem]

[0014] In order to solve the above-mentioned problems, a spatial audio transmission device according to an embodiment of the present disclosure includes a plurality of servers that are communicatively connected to a plurality of client terminals that execute multi-player real-time content and are separated into respective multicast groups, and a server included in each multicast group, when receiving first player information from the client terminal that executes the multi-player real-time content, transmits the first player information to other servers included in the multicast group by multicast to share the first player information, and when receiving second player information from other servers included in the multicast group, mixes spatial audio based on the first player information and the second player information and transmits the mixed audio to the client terminal by unicast.

[0015] In an embodiment, when the plurality of servers are separated according to the multicast groups, servers that are communicatively connected to each other via the same network are grouped into one multicast group.

[0016] In an embodiment, when the server receives the first player information, the server receives first player information located in a three-dimensional virtual space from a first client terminal that executes the multi-player real-time content.

[0017] In an embodiment, the first player information includes position information, direction information and voice information of the first player.

[0018] In an embodiment, when receiving the first player information, the server receives the first player information located in a three-dimensional virtual space from the first client terminal in a unicast manner.

[0019] In an embodiment, when the server mixes and transmits the spatial audio, when the first player information is received, the server separates encoded voice information of the first player and position and direction information of the first player from the first player information, and when the second player information is received, the server separates encoded voice information of the second player and position and direction information of the second player from the second player information, decodes the encoded voice information of the first player and the voice information of the second player to generate raw audio data, mixes the spatial audio based on the position and direction information of the first player and the second player and the raw audio data to generate stereo audio data, and encodes the stereo audio data to transmit compressed audio.

[0020] In an embodiment, the server performs real-time rendering based on position and direction information of the first player and the second player to generate raw picture data, and encodes the raw picture data to transmit compressed video.

[0021] In an embodiment, the server includes a unicast receiving unit that receives first player information from the client terminal in a unicast manner, a multicast receiving unit that receives second player information from the other server in a multicast manner, an audio decoder unit that decodes encoded audio information of the first player separated from the first player information and encoded audio information of the second player separated from the second player information to output raw audio data, a spatial audio engine unit that mixes spatial audio based on position and direction information of the first player separated from the first player information, position and direction information of the second player separated from the second player information, and the raw audio data to output stereo audio data, an audio encoder unit that encodes the stereo audio data to output compressed audio, a multicast transmitting unit that transmits the first player information to other servers in a multicast manner, and a unicast transmitting unit that transmits the compressed audio to the client terminal in a unicast manner.

[0022] In an embodiment, the server includes a real-time rendering unit that performs real-time rendering processing based on position and direction information of the first player separated from the first player information and position and direction information of the second player separated from the second player information, and outputs raw picture data, and a video encoder that encodes the raw picture data and outputs compressed video.

[0023] A spatial audio transmission method according to an embodiment of the present disclosure is a spatial audio transmission method for an apparatus including a plurality of servers communicatively connected to a plurality of client terminals and separated for each multicast group, the method including the steps of receiving first player information from the client terminal, multicasting the first player information to other servers included in the multicast group to share the first player information, receiving second player information from other servers included in the multicast group, mixing spatial audio based on the first player information and the second player information, and unicasting the mixed spatial audio to the client terminal.

[0024] A computer program for providing a spatial audio transmission method according to another embodiment of the present disclosure for solving the above-mentioned problems is stored on a medium for causing a hardware computer to execute any one of the above-mentioned methods.

[0025] In addition, other methods and systems for implementing the present disclosure, and a computer-readable recording medium having a computer program for executing the methods may also be provided. Effect of the Invention

[0026] As described above, according to the present disclosure, player information of multi-player real-time content received from a client terminal is shared by multicasting it to a server included in a multicast group, and spatial audio data is mixed based on the shared player information, thereby eliminating transmission of duplicated data, reducing the load on the network, and enabling efficient transmission of spatial audio data.

[0027] That is, the present disclosure uses multicast instead of unicast, thereby eliminating duplicate data transmissions and reducing the load on the network.

[0028] Furthermore, in the present disclosure, since it is sufficient that the server exists within the same network, the client terminal can be independent of its physical location.

[0029] The effects of the present disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a diagram illustrating a spatial audio transmission device according to the present disclosure. [Diagram 2] 2 is a diagram for explaining player information of multi-player real-time content executed on the client terminal of FIG. 1. [Diagram 3] FIG. 2 is a diagram for explaining the operation process of the servers included in each multicast group in FIG. [Figure 4] FIG. 2 is a block diagram illustrating the configuration of a server in FIG. [Diagram 5] FIG. 2 is a block diagram illustrating the configuration of a server in FIG. [Figure 6] 1 is a flowchart illustrating a spatial audio transmission method according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The advantages and features of the present invention, and the methods for achieving them, will become clear from the following detailed description of the embodiments in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, and may be realized in various different forms. However, the embodiments are provided to complete the disclosure of the present disclosure and to allow those skilled in the art to fully understand the scope of the present disclosure, and the present disclosure is only defined by the scope of the claims.

[0032] The terms used in this specification are for the purpose of describing the embodiments and are not intended to limit the present disclosure. In this specification, the singular includes the plural unless otherwise specified. The terms "comprises" and / or "comprising" used in the specification do not exclude the presence or addition of one or more other components other than the components mentioned. The same reference numerals throughout the specification refer to the same components, and "and / or" includes each and every combination of one or more of the components mentioned. Even if "first", "second", etc. are used to describe various components, it is natural that these components are not limited by these terms. These terms are used simply to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below can also be the second component within the technical spirit of the present disclosure.

[0033] Unless otherwise defined, all terms (including technical and scientific terms) used in this specification are used as meanings commonly understood by ordinary skilled artisans in the technical field to which the present invention belongs. Furthermore, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly and specifically defined.

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0035] Prior to the description, the meanings of the terms used in this specification will be briefly explained. However, since the explanations of the terms are intended to facilitate understanding of this specification, it should be noted that unless explicitly stated as matters limiting this disclosure, they are not used to limit the technical ideas of this disclosure.

[0036] FIG. 1 is a diagram illustrating a spatial audio transmission device according to the present disclosure.

[0037] As shown in FIG. 1, the spatial audio transmission device according to the present disclosure may include a plurality of servers 100 that are communicatively connected to a plurality of client terminals 200 executing multi-player real-time content via a network and are separated into multicast groups 300.

[0038] Here, when the multiple servers 100 are separated into each multicast group 300, the servers 100 that are communicatively connected to each other on the same network can be grouped into one multicast group 300.

[0039] As an example, as shown in FIG. 1, the multiple servers 100 can be divided into a first multicast group 310 and a second multicast group 320, but this is merely an embodiment and is not limited thereto.

[0040] Here, the servers 100 included in the first multicast group 310 are communicatively connected to a first network, and the servers 100 included in the second multicast group 320 are communicatively connected to a second network.

[0041] The client terminal 200 may include any of standing devices such as a PC (Personal Computer), a Network TV, a Hybrid Broadcast Broadband TV (HBTV), a Smart TV, an IPTV (Internet Protocol TV), etc., and mobile devices or handheld devices such as a smart phone, a tablet PC, a notebook, a PDA (Personal Digital Assistant), etc.

[0042] Next, the network that communicatively connects the client terminal 200 and the server 100 includes both wired and wireless networks, but is generally referred to as a communication network that supports various communication standards or protocols for pairing and / or data transmission and reception between the client terminal 200 and the server 100.

[0043] Such wired / wireless networks include any communication network currently or hereafter supported by a standard, and may support one or more communication protocols therefor.

[0044] Such wired / wireless networks include, for example, networks for wired connections such as USB (Universal Serial Bus), CVBS (Composite Video Banking Sync), Component, S-Video (analog), DVI (Digital Visual Interface), HDMI (High Definition Multimedia Interface), RGB, and D-SUB, and communication standards or protocols therefor, as well as Bluetooth (Bluetooth (registered trademark)), RFID (Radio Frequency Identification), infrared communication (IrDA: infrared Data Association), UWB (Ultra Wideband), ZigBee (ZigBee (registered trademark)), DLNA (Digital Living Network Alliance), WLAN (Wireless LAN) (Wi-Fi (registered trademark)), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), LTE / LTE-A (Long Term The wireless communication network may be formed by a network for wireless connection such as LTE (LTE Evolution / Advanced) and Wi-Fi (registered trademark) direct, and a communication standard or protocol therefor.

[0045] Next, when the server 100 included in each multicast group 300 receives first player information from the client terminal 200 executing the multi-player real-time content, the server 100 multicasts and shares the first player information to the remaining servers 100 included in the multicast group 300, and when the server 100 receives second player information from another server included in the multicast group 300, the server 100 mixes spatial audio based on the first player information and the second player information and unicasts the mixed audio to the client terminal 200.

[0046] Here, when receiving the first player information, the server 100 may receive the first player information located within the three-dimensional virtual space from a first client terminal executing the multi-player real-time content.

[0047] As an example, the first player information may include position information, direction information, and voice information of the first player, but this is merely an example and is not limited thereto.

[0048] When receiving the first player information, the server 100 can receive the first player information located in the three-dimensional virtual space from the first client terminal in a unicast manner.

[0049] Next, when the server 100 transmits the first player information by multicast, upon receiving the first player information, the server 100 acquires the group address of the multicast group 300, and transmits the first player information to the remaining servers in the multicast group 300 by multicast based on the acquired group address.

[0050] Here, when the server 100 receives the first player information, it stores the first player information in a database of the server 100, and then transmits the first player information to the remaining servers in the multicast group 300 in a multicast manner, so that all servers in the multicast group 300 can share the first player information.

[0051] Next, when the server 100 receives the second player information, it can receive the second player information from any one of the remaining other servers included in the multicast group 300 in a multicast manner.

[0052] Here, when another server receives second player information from a second client terminal executing multi-player real-time content, the server can transmit the second player information to the remaining servers included in the multicast group 300 in a multicast manner to share the information.

[0053] In this case, when the other server receives the second player information, if the second client terminal executes the multi-player real-time content, the other server can receive the second player information located in the three-dimensional virtual space from the second client terminal.

[0054] As an example, the second player information may include position information, direction information, and voice information of the second player, but this is merely an example and is not limited thereto.

[0055] Furthermore, when receiving the second player information, the other server can receive the second player information located in the three-dimensional virtual space from the second client terminal in a unicast manner.

[0056] Then, when the other servers transmit the second player information by multicast, upon receiving the second player information, they can acquire the group address of the multicast group 300, and transmit the second player information to the remaining other servers in the multicast group by multicast based on the acquired group address.

[0057] Here, when the other server receives the second player information, it stores the second player information in its database, and then transmits the second player information to the remaining other servers in the multicast group 300 in a multicast manner, so that all servers in the multicast group 300 can share the second player information.

[0058] Next, when the server 100 receives first player information from the client terminal 200 in a unicast manner, it can store the first player information in the database of the server 100, and when the server 100 receives second player information from another server in a multicast manner, it can store the second player information in the database of the server 100.

[0059] Next, when the server 100 mixes and transmits the spatial audio, when first player information is received, the server 100 separates the encoded voice information of the first player and the position and direction information of the first player from the first player information, and when second player information is received, the server 100 separates the encoded voice information of the second player and the position and direction information of the second player from the second player information, decodes the encoded voice information of the first player and the voice information of the second player to generate raw audio data, mixes the spatial audio based on the position and direction information of the first player and the second player and the raw audio data to generate stereo audio data, and encodes the stereo audio data to transmit compressed audio.

[0060] Here, when transmitting compressed audio, the server 100 can transmit the compressed audio to the client terminal 200 in a unicast manner.

[0061] In some cases, the server 100 may perform real-time rendering based on the position and orientation information of the first and second players to generate raw picture data, and encode the raw picture data to transmit compressed video.

[0062] Here, when transmitting the compressed video, the server 100 can transmit the compressed video to the client terminal 200 in a unicast manner.

[0063] On the other hand, multi-player real-time content executed on the client terminal 200 is content that supports spatial audio, enabling players to have a more vivid experience.

[0064] Unlike pre-recorded content such as movies, multi-player real-time content requires spatial audio to be generated in real time as the distance and direction between the players and the sound source change in real time.

[0065] As an example, the multiplayer real-time content may be 3D virtual content such as a metaverse, and players of the 3D virtual content may be characters within a 3D virtual space.

[0066] Therefore, the present disclosure enables the client terminal 200 to receive player information located within the three-dimensional virtual space of the multi-player real-time content in a unicast manner.

[0067] Here, the player information may include position information, direction information, and voice information of the player, but this is merely an example and is not limited thereto.

[0068] In addition, the present disclosure relates to an apparatus including a system for transmitting content that changes in real time in response to user interaction, such as a game, unlike simple audio / video streaming, and can transmit data that reflects the current state in real time, unlike pre-recorded audio / video streaming.

[0069] Next, the present disclosure can include an audio engine that provides spatial audio. The audio engine is software that mixes audio data input from multiple channels and outputs it to an output channel (e.g., 2-channel stereo), and can output audio with a sense of space by inputting position and direction data of a sound source in addition to audio data to the input audio channel.

[0070] In addition, the present disclosure allows client terminal 200 to receive player information of multi-player real-time content in a unicast manner. The unicast manner is a one-to-one transmission manner in which data sent by one sender is received by one receiver, and when N players share their own status data via unicast, a total of N (N-1) data transmissions are required.

[0071] In the present disclosure, player information of multi-player real-time content received from a client terminal 200 can be shared by multicasting it to a server 100 included in a multicast group 300. A multicast method is a method in which one sender transmits data one-to-many to all receivers belonging to a multicast group. Unlike a unicast method, the multicast method does not transmit the same data separately to each receiver, but transmits one piece of data to a multicast group, and all receivers belonging to the group can receive the transmitted data. Therefore, when N players can share their own state data via multicast, data can be transmitted only N times in total, which is efficient.

[0072] In this way, the present disclosure groups multiple audio streaming servers into one multicast group on the same network, and transmits shared data between the servers by multicast, thereby reducing the load on the network by using multicast instead of unicast, and eliminating duplicate data transmission. Also, since it is only necessary for the servers to be present in the same network, the client terminals can be free from the physical location.

[0073] In this manner, the present disclosure enables efficient transmission of spatial audio data by eliminating transmission of duplicated data, reducing the load on the network, and multicasting and sharing player information of multi-player real-time content received from client terminals to servers included in a multicast group, and mixing spatial audio data based on the shared player information.

[0074] FIG. 2 is a diagram for explaining player information of multi-player real-time content executed on the client terminal of FIG.

[0075] As shown in FIG. 2, the multiplayer real-time content executed on the client terminal may be content that supports spatial audio.

[0076] As an example, the multiplayer real-time content is content having a three-dimensional virtual space 500 such as a metaverse, and the players 400 in the three-dimensional virtual space 500 may be characters in the three-dimensional virtual space 500.

[0077] Unlike pre-recorded content such as movies, multi-player real-time content requires spatial audio to be generated in real time as the distance and direction between the players and the sound source change in real time.

[0078] Therefore, the present disclosure enables information about players 400 located within a three-dimensional virtual space 500 of multi-player real-time content to be received from a client terminal in a unicast manner.

[0079] Here, the player 400 information may include position information, direction information, and voice information of the player 400, but this is merely an example and is not limited thereto.

[0080] As shown in FIG. 2, when a first player 410, a second player 420, and a third player 430 exist within a three-dimensional virtual space 500, the first player 410 may be located closest to the second player 420 and the third player 430, and the voice 412 of the first player 410 may be the loudest, the second player 420 may be located farther away than the first player 410 and closer than the third player 430, and the voice 422 of the second player 420 may be quieter than the first player 410 and louder than the third player 430, and the third player 430 may be located farthest away from the first player 410 and the second player 420, and the voice 432 of the third player 430 may be the quietest.

[0081] Therefore, the present disclosure can mix spatial audio based on position information, direction information and audio information of multiple players 400 in a three-dimensional virtual space and efficiently transmit the mixed audio to a client terminal.

[0082] FIG. 3 is a diagram for explaining the operation process of the servers included in each multicast group in FIG.

[0083] As shown in FIG. 3, the present disclosure allows multiple servers that are communicatively connected to one another via the same network to be grouped into one multicast group 300.

[0084] As an example, if one multicast group 300 includes a first server 110, a second server 120, and a third server 130, the first server 110 can unicast first player information including the position, direction, and voice of the first player from a first client terminal 210 that is running multi-player real-time content, and the second server 120 can unicast second player information including the position, direction, and voice of the second player from a second client terminal 220 that is running multi-player real-time content.

[0085] The first server 110 then multicasts the first player information, including the position, direction, and voice of the first player, to the multicast group address, so that the first player information can be multicast and shared to the second server 120 and the third server 130 in the multicast group 300.

[0086] In addition, the second server 120 multicasts second player information including the position, direction, and voice of the second player to the multicast group address, so that the second player information can be multicast and shared to the first server 110 and the third server 130 in the multicast group 300.

[0087] Next, the third server 130 receives the first player information and the second player information via multicast, and when it receives the third player information including the position, direction, and voice of the third player via unicast from the third client terminal 230 executing the multi-player real-time content, it combines the first player information, the second player information, and the third player information to mix spatial audio, and can unicast the mixed spatial audio to the third client terminal 230.

[0088] In addition, when the third server 130 receives third player information including the position, direction, and voice of the third player by unicast from the third client terminal 230, the third server 130 multicasts the third player information including the position, direction, and voice of the third player to the multicast group address, so that the third player information can be multicast and shared to the first server 110 and the second server 120 in the multicast group 300.

[0089] In this way, the server of the present disclosure can receive player information of multi-player real-time content from the client terminal by unicast, and can transmit mixed spatial audio to the client terminal by unicast.

[0090] Here, the unicast method is a one-to-one transmission method in which data sent by one sender is received by one receiver. If N players share their status data via unicast, a total of N(N-1) data transmissions are required.

[0091] The server of the present disclosure can share player information of multi-player real-time content received from client terminals by multicasting it to servers included in the multicast group.

[0092] Here, the multicast method is a method in which one sender transmits data one-to-many to all receivers belonging to a multicast group. Unlike unicast, the same data is not transmitted separately to each receiver. Instead, when one piece of data is transmitted to a multicast group, all receivers belonging to the group can receive the transmitted data. Therefore, when N players share their status data through multicast, data can be transmitted only N times in total, which is efficient.

[0093] 4 and 5 are block diagrams illustrating the server of FIG.

[0094] As shown in FIG. 4, a server 100 according to one embodiment of the present disclosure may include a unicast receiver unit 101, a multicast receiver unit 102, an audio decoder unit 103, a spatial audio engine unit 104, an audio encoder unit 105, a multicast transmitter unit 106, and a unicast transmitter unit 107.

[0095] Here, the unicast receiving unit 101 can receive the first player information from the client terminal by unicast (A).

[0096] As an example, the first player information may include position information, orientation information, and voice information of the first player.

[0097] Then, the multicast receiver 102 can receive second player information from other servers in the multicast group by multicasting (B).

[0098] As an example, the second player information may include position information, orientation information, and voice information of the second player.

[0099] Next, the audio decoder unit 103 can decode the encoded audio information of the first player separated from the first player information and the encoded audio information of the second player separated from the second player information, and output raw audio data.

[0100] Next, the spatial audio engine unit 104 can mix the spatial audio based on the position and direction information of the first player separated from the first player information, the position and direction information of the second player separated from the second player information, and the raw audio data, and output stereo audio data.

[0101] Here, the spatial audio engine unit 104 is software that mixes audio data input from multiple channels and outputs it to an output channel (e.g., 2-channel stereo). By inputting sound source position and direction data in addition to audio data to the input audio channel, audio with a sense of space can be output.

[0102] The audio encoder unit 105 can encode stereo audio data and output compressed audio.

[0103] Next, the multicast transmission unit 106 can transmit the first player information to other servers in the multicast group by multicasting (C).

[0104] Next, the unicast transmission unit 107 can transmit the compressed audio to the client terminal in a unicast manner (D).

[0105] Also, as shown in FIG. 5, a server 100 according to another embodiment of the present disclosure may include a unicast receiving unit 101, a multicast receiving unit 102, an audio decoder unit 103, a spatial audio engine unit 104, an audio encoder unit 105, a multicast transmitting unit 106, a unicast transmitting unit 107, a real-time rendering unit 108, and a video encoder unit 109.

[0106] Here, the unicast receiving unit 101 can receive the first player information from the client terminal by unicast (A).

[0107] As an example, the first player information may include position information, orientation information, and voice information of the first player.

[0108] Then, the multicast receiver 102 can receive second player information from other servers in the multicast group by multicasting (B).

[0109] As an example, the second player information may include position information, orientation information, and voice information of the second player.

[0110] Next, the audio decoder unit 103 can decode the encoded audio information of the first player separated from the first player information and the encoded audio information of the second player separated from the second player information, and output raw audio data.

[0111] Next, the spatial audio engine unit 104 can mix the spatial audio based on the position and direction information of the first player separated from the first player information, the position and direction information of the second player separated from the second player information, and the raw audio data, and output stereo audio data.

[0112] Here, the spatial audio engine unit 104 is software that mixes audio data input from multiple channels and outputs it to an output channel (e.g., 2-channel stereo). By inputting sound source position and direction data in addition to audio data to the input audio channel, audio with a sense of space can be output.

[0113] The audio encoder unit 105 can then encode the stereo audio data and output compressed audio.

[0114] Next, the multicast transmission unit 106 can transmit the first player information to other servers in the multicast group by multicasting (C).

[0115] The real-time rendering unit 108 can perform real-time rendering processing based on the position and direction information of the first player separated from the first player information and the position and direction information of the second player separated from the second player information, and output raw picture data.

[0116] Next, the video encoder unit 109 can encode the raw image data and output compressed video.

[0117] Next, the unicast transmission unit 107 can transmit the compressed audio and compressed video to the client terminal in a unicast manner (D).

[0118] FIG. 6 is a flowchart illustrating a spatial audio transmission method according to the present disclosure.

[0119] As shown in FIG. 6, in the present disclosure, first player information can be received by unicast from a client terminal (S10).

[0120] Here, in the present disclosure, first player information located within a three-dimensional virtual space can be received from a first client terminal executing multi-player real-time content.

[0121] As an example, the first player information may include position information, direction information, and voice information of the first player.

[0122] Then, in the present disclosure, the first player information can be shared by being multicast to the remaining other servers included in the multicast group (S20).

[0123] Here, in the present disclosure, when first player information is received, a group address of a multicast group is acquired, and the first player information can be transmitted to other remaining servers in the multicast group in a multicast manner based on the acquired group address.

[0124] As an example, in the present disclosure, upon receiving first player information, the first player information is stored in a database, and then the first player information is transmitted to the remaining servers in the multicast group in a multicast manner, so that all servers in the multicast group can share the first player information.

[0125] Next, in the present disclosure, second player information can be received by multicast from other servers included in the multicast group (S30).

[0126] Here, the present disclosure can receive the second player information from any one of the remaining other servers included in the multicast group in a multicast manner.

[0127] As an example, the second player information may include position information, direction information, and voice information of the second player.

[0128] Next, in the present disclosure, the spatial audio can be mixed based on the first player information and the second player information (S40).

[0129] Here, in the present disclosure, when first player information is received, encoded audio information of the first player and position and direction information of the first player are separated from the first player information, and when second player information is received, encoded audio information of the second player and position and direction information of the second player are separated from the second player information, the encoded audio information of the first player and the audio information of the second player are decoded to generate raw audio data, spatial audio is mixed based on the position and direction information of the first player and the second player and the raw audio data to generate stereo audio data, and the stereo audio data is encoded to output compressed audio.

[0130] Then, in the present disclosure, the mixed spatial audio can be unicast transmitted to the client terminal (S50).

[0131] In addition, in the present disclosure, raw picture data can be generated by performing real-time rendering processing based on position and direction information of the first player and the second player, and the raw picture data can be encoded to output compressed video.

[0132] The present disclosure then allows for the compressed video to be transmitted to the client terminal in a unicast manner.

[0133] According to an embodiment of the present disclosure, the system server 100 may be configured with one or more cores and may include a processor for data analysis and deep learning, such as a central processing unit (CPU) of a computing device, a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU). The system server 100 may read a computer program stored in a database and perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the system server 100 may perform calculations for learning a neural network. The system server 100 may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from the input data, calculating errors, and updating weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the system server 100 may process learning of a network function. For example, the CPU and the GPGPU may both process learning of a network function and data classification using the network function. In addition, in one embodiment of the present disclosure, the processors of the multiple computing devices can be used together to process the learning of the network function and the data classification using the network function. In addition, the computer program executed in the computing device according to one embodiment of the present disclosure can be a CPU, GPGPU, or TPU executable program.

[0134] According to an embodiment of the present disclosure, the database can store computer programs for performing video and audio streaming, and the stored computer programs can be read and run by the processor. The database can store any type of information generated or determined by the system server 100 and any type of information received by the network unit.

[0135] According to an embodiment of the present disclosure, the database may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), magnetic memory, magnetic disk, and optical disk. The system server 100 may also operate in association with a web storage that performs the storage function of the database on the Internet. The above description of the database is merely exemplary and is not limiting.

[0136] The system server 100 of the present disclosure may further include an output unit and an input unit.

[0137] The output unit according to an embodiment of the present disclosure may display a user interface (UI) for providing video and audio streaming results. The output unit may output any type of information generated or determined by the system server 100 and any type of information received.

[0138] In an embodiment of the present disclosure, the output unit may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, and a 3D display. Some of these display modules may be configured as a transparent type or a light-transmitting type so that the outside can be seen through the display module. This may be called a transparent display module, and a representative example of the transparent display module is a TOLED (Transparent OLED).

[0139] The input unit according to an embodiment of the present disclosure can receive a user's input. The input unit can include keys and / or buttons on a user interface for receiving the user's input, or physical keys and / or buttons. The user's input via the input unit can execute a computer program for controlling a display according to an embodiment of the present disclosure.

[0140] The input unit according to the embodiment of the present disclosure may receive a signal by sensing a button operation or touch input by a user, or may receive a voice or a motion of a user through a camera or microphone and convert it into an input signal. For this purpose, a speech recognition technology or a motion recognition technology may be used.

[0141] The input unit according to the embodiment of the present disclosure may be realized as an external input device connected to the system server 100. For example, the input device may be at least one of a touchpad, a touch pen, a keyboard, or a mouse for receiving a user's input, but this is merely an example and is not limited thereto.

[0142] The input unit according to an embodiment of the present disclosure can recognize a touch input by a user. The input unit according to an embodiment of the present disclosure can have the same configuration as the output unit. The input unit can be configured as a touch screen that is realized to receive a selection input by a user. The touch screen can be any one of a contact type capacitive type, an infrared light sensing type, a surface acoustic wave (SAW) type, a piezoelectric type, and a resistive film type. The detailed description of the touch screen described above is merely an example according to an embodiment of the present disclosure, and various touch screen panels can be adopted in the system server 100. The input unit configured as a touch screen can include a touch sensor. The touch sensor can be configured to convert a pressure applied to a specific portion of the input unit or a change in capacitance generated at a specific portion of the input unit into an electrical input signal. The touch sensor can be configured to detect not only the position and area of ​​the touch, but also the pressure at the time of touch. When there is a touch input to the touch sensor, a signal corresponding thereto is sent to the touch controller. The touch controller can process the signal and then transmit corresponding data to the system server 100. This allows the system server 100 to recognize which area of ​​the input unit has been touched.

[0143] In this manner, in the present disclosure, player information of multi-player real-time content received from client terminals is shared by multicast transmission to servers included in the multicast group, and spatial audio data is mixed based on the shared player information, thereby eliminating transmission of duplicated data, reducing the load on the network, and enabling efficient transmission of spatial audio data.

[0144] That is, in the present disclosure, multicast is used instead of unicast, so that duplicate data transmission can be eliminated and the load on the network can be reduced.

[0145] Furthermore, in the present disclosure, since it is sufficient that the server exists within the same network, the client terminal can be independent of its physical location.

[0146] The method according to the embodiment of the present disclosure described above can be realized as a program (or application) to be executed by a server, which is hardware, and stored on a medium.

[0147] The above-mentioned program may include code coded in a computer language such as C, C++, JAVA, machine language, etc., which is read by the processor (CPU) of the computer through the device interface of the computer so that the computer can read the program and execute the method realized by the program. Such code may include functional code related to functions defining functions necessary to execute the method, and may include control code related to execution procedures necessary for the processor of the computer to execute the function according to a predetermined procedure. In addition, such code may further include memory reference-related code for referring to a location (address) of the internal or external memory of the computer where additional information or media necessary for the processor of the computer to execute the function should be referenced. Furthermore, if the processor of the computer needs to communicate with any other remote computer or server to execute the function, the code may further include communication-related code for how to communicate with any other remote computer or server using a communication module of the computer, and what information or media should be sent or received during communication.

[0148] The storage medium means a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short time such as a register, cache, memory, etc. Specific examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data recording device, etc. That is, the program can be stored in various recording media on various servers to which the computer can be connected or various recording media on the user's computer. Also, the medium can be distributed among computer systems connected by a network and store computer-readable code in a distributed manner.

[0149] The steps of a method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, or in a software module executed by hardware, or in a combination thereof. The software module may be present in a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Flash Memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer readable recording medium known in the art to which the present disclosure pertains.

[0150] Although the embodiments of the present disclosure have been described above with reference to the accompanying drawings, those skilled in the art to which the present disclosure pertains should understand that the present disclosure can be embodied in other specific forms without changing the technical spirit or essential features thereof. Therefore, the above-described embodiments should be understood to be illustrative in all respects and not restrictive.

Claims

1. The system includes a plurality of servers that are communicatively connected to a plurality of client terminals that execute multi-player real-time content and are separated for each multicast group; The servers included in each of the multicast groups are: a spatial audio transmission device which, when receiving first player information from a client terminal executing the multi-player real-time content, transmits the first player information to remaining servers included in the multicast group by multicast to share the first player information, and when receiving second player information from another server included in the multicast group, mixes spatial audio based on the first player information and the second player information and transmits the mixed audio by unicast to the client terminal.

2. The plurality of servers include The spatial audio transmission device according to claim 1 , wherein when the servers are separated into the multicast groups, the servers that are communicatively connected to each other via the same network are grouped into one multicast group.

3. The server, 2. The spatial audio transmission device according to claim 1, wherein when the first player information is received, the first player information located in a three-dimensional virtual space is received from a first client terminal that executes the multi-player real-time content.

4. The first player information is The spatial audio transmission device according to claim 3 , further comprising position information, direction information and voice information of the first player.

5. The server, 4. The spatial audio transmission device according to claim 3, wherein when the first player information is received, the first player information located in the three-dimensional virtual space is received from the first client terminal by a unicast method.

6. The server, When the spatial audio is mixed and transmitted, when the first player information is received, encoded voice information of the first player and position and direction information of the first player are separated from the first player information; When the second player information is received, separating the encoded second player voice information and the second player position and direction information from the second player information; Decoding the encoded first player voice information and second player voice information to generate raw audio data; generating stereo audio data by mixing spatial audio based on position and direction information of the first player and the second player and the low audio data; 2. The spatial audio transmission device according to claim 1, wherein the stereo audio data is encoded and compressed audio is transmitted.

7. The server, generating raw picture data by performing real-time rendering processing based on position and direction information of the first player and the second player; 7. The spatial audio transmission device according to claim 6, wherein the raw image data is encoded and compressed video is transmitted.

8. The server, a unicast receiving unit that receives first player information from the client terminal by a unicast method; a multicast receiving unit that receives second player information from the other server by multicast; an audio decoder unit for decoding the encoded audio information of the first player separated from the first player information and the encoded audio information of the second player separated from the second player information to output raw audio data; a spatial audio engine unit that mixes spatial audio based on position and direction information of the first player separated from the first player information, position and direction information of the second player separated from the second player information, and the low audio data, and outputs stereo audio data; an audio encoder unit that encodes the stereo audio data and outputs compressed audio; a multicast transmission unit that transmits the first player information to other servers by multicast; a unicast transmission unit for transmitting the compressed audio to the client terminal in a unicast manner; 2. The spatial audio transmission device according to claim 1, further comprising:

9. The server, a real-time rendering unit that performs real-time rendering based on position and direction information of the first player separated from the first player information and position and direction information of the second player separated from the second player information, and outputs raw picture data; a video encoder unit that encodes the raw image data and outputs a compressed video; 9. The spatial audio transmission device according to claim 8, further comprising:

Images