Distribution system

Abstract

To distribute proper sound to a client.SOLUTION: A distribution system 1 executes first distribution processing to create sound distributed to one client and corresponding to one of plural positions in a three-dimensional space, and second distribution processing to distribute sound source information relating to one sound source and containing positional information in the three-dimensional space of the sound source with different processes, and changes a ratio between a process for executing the first distribution processing and a process for executing the second distribution processing based on a predetermined reference. The ratio may be changed based on at least one of the number of clients and the number of sound sources.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] One aspect of the present disclosure relates to a distribution system for distributing video and audio. [Background technology]

[0002] The following Patent Document 1 discloses a distributed game engine having multiple game engine nodes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-510600 Summary of the Invention [Problem to be solved by the invention]

[0004] General game engines are designed to output only one audio channel. Therefore, even if video from multiple camera viewpoints is streamed (distributed) on one game engine, the audio will not be tailored to the camera viewpoints, but will be a single channel common to all, and it is not possible to deliver (multiple) audio tailored to each user position (multiple camera viewpoints). In other words, it is not possible to deliver appropriate audio to the client. Therefore, there is a demand for delivering appropriate audio to the client. [Means for solving the problem]

[0005] A distribution system according to one aspect of the present disclosure is a distribution system that executes, in different processes, a first distribution process that generates audio corresponding to one of a plurality of positions in three-dimensional space and distributes the audio to a single client, and a second distribution process that distributes sound source information related to a single sound source, the sound source information including position information of the single sound source in three-dimensional space, to each client, and changes the ratio between the process that executes the first distribution process and the process that executes the second distribution process based on a predetermined criterion.

[0006] In such an aspect, in the process in which the first distribution process is executed, a sound corresponding to one position in three-dimensional space is generated and distributed to one client. Therefore, it is possible to distribute appropriate sound to the client. Also, in the process in which the second distribution process is executed, sound source information related to one sound source, which includes position information of the one sound source in three-dimensional space, is distributed to each client. For example, each client can play appropriate sound based on the distributed sound source information. Therefore, it is possible to distribute appropriate sound to the client. Effect of the Invention

[0007] According to one aspect of the present disclosure, appropriate audio can be delivered to a client. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing a system configuration of a conventional distribution system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a system configuration of a distribution system (process B type) according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a system configuration of a distribution system (process C type) according to an embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a positional relationship in a virtual space. [Diagram 5] 13 is a flowchart illustrating an example of a ratio change process executed by the distribution system according to the embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a system configuration before a ratio change in a distribution system according to an embodiment. [Figure 7] FIG. 13 is a diagram illustrating an example of a system configuration after a ratio change in the distribution system according to the embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer used in the distribution system according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated descriptions will be omitted. In addition, the embodiments of the present disclosure in the following description are specific examples of the present invention, and the present invention is not limited to these embodiments unless otherwise specified to limit the present invention.

[0010] First, a description will be given of the virtual / game space and the like that are the subject of the embodiment. It is assumed that the service that is the subject of the embodiment is composed of a system (server) that generates and distributes images of the virtual space or game screens, and an unspecified number of clients that receive the images. The clients can transmit operation signals to the virtual space, and avatars (characters) that correspond to the clients in the virtual space can be controlled by the operation signals (a service based on a technique known as cloud gaming).

[0011] For virtual spaces, we assume a game engine that can generate multiple systems of images using computer graphics (CG), but has limitations when it comes to generating multiple systems of sound. However, as one system (or a small number of systems less than the number of images), stereo or surround sound can be generated, and a three-dimensional sound effect can be realized that is synthesized from background music (or sounds used regardless of position, sound effects, announcements, etc.) placed in the virtual space, or multiple sound sources (sounds generated in relation to a position in the space, such as the sound of a waterfall or the sound of glass breaking) within the virtual space.

[0012] For example, in order for multiple clients (users) to operate avatars from a first-person viewpoint, images from multiple viewpoints are generated, and then realistic audio is generated for each position, but if only one audio system can be handled, it is not possible to generate audio corresponding to the number of images. Note that in the embodiments, the term "audio" may be replaced with "sound."

[0013] Fig. 1 is a diagram showing the system configuration of a conventional distribution system, which streams (distributes) video and audio to three clients, namely, client α, client β, and client γ.

[0014] On the server (conventional distribution system) side, four game engine processes are running. Three of the four processes are for each client; that is, a process for client α (hereafter referred to as "α process"), a process for client β (hereafter referred to as "β process"), and a process for client γ (hereafter referred to as "γ process"). The remaining one of the four processes is a process (hereafter referred to as the "synchronization process") that manages the synchronization of the processes for each client mentioned above. Each process exists in the same environment, and (the synchronization process takes the lead) synchronizes processing between the processes.

[0015] The α process, β process, and γ process each execute CG generation processing, synchronization processing, video processing, audio processing, and calculation processing of interactions in virtual space. The CG generation processing generates CG for the target client (for example, client α for the α process). The synchronization processing works in conjunction with the synchronization process to perform synchronization between the three processes for each client. The video processing generates video for the target client based on CG generated by the CG generation processing in the same process. The audio processing generates audio for the target client. The calculation processing of interactions in virtual space calculates interactions such as between avatars in the virtual space. It should be noted that the CG generation processing, synchronization processing, video processing, and audio processing may be based on the calculation results of interactions in virtual space.

[0016] Each of the α process, β process, and γ process streams the video generated by processing the video of the process and the audio generated by processing the audio to the target client. The α process, β process, and γ process each perform calculation processing of interactions in a virtual space by receiving operation signals such as an avatar from the target client and performing calculations based on the received operation signals.

[0017] The sound assets include background music, sound source 1, sound source 2, etc., and are stored in the server's storage. The sound assets are loaded into the processes that require them as necessary.

[0018] One of the challenges of the system configuration of the conventional distribution system shown in Figure 1 is that even within a single server, each process needs to perform synchronization between processes and calculations of interactions in a common virtual space.

[0019] The delivery system (delivery system 1) according to the embodiment can have two types of system configurations, either exclusively or non-exclusively (used simultaneously). The first type of system configuration is referred to as process B type, and the second type of system configuration is referred to as process C type. Below, first, the details of the process B type and then the details of the process C type will be described in order.

[0020] [Process Type B] Process B type of distribution system 1 executes multiple processes for processing at least one of video and audio, and executes, in separate processes, a common process (executed in process A described below) for generating video corresponding to each of multiple positions in three-dimensional space and distributing it to each client, and a distribution process (executed in process B described below) for generating audio corresponding to one of the multiple positions and distributing it to a single client.

[0021] The three-dimensional space may be a virtual space. The process for processing audio may only be capable of processing one line of audio. The one position may be a position where an avatar corresponding to one client is located. The common process and multiple distribution processes for multiple clients may be executed by different processes. The common process may distribute the generated video on a one-to-many basis. The common process may distribute the generated video via an SFU (Selective Forwarding Unit). The number of processes for executing the distribution process may be based on the number of clients. For example, the number of processes for executing the distribution process may be the same as the number of clients, may be equal to or greater than the number of clients, or may dynamically increase or decrease according to the number of clients. The distribution process does not have to perform processing related to video.

[0022] Fig. 2 is a diagram showing an example of a system configuration of the distribution system 1 (process B type). In the following description of Fig. 2, description of the same contents as in Fig. 1 will be omitted as appropriate.

[0023] The distribution system 1 (process B type) is composed of multiple game engine processes that operate independently on one or multiple servers. Specifically, the distribution system 1 (process B type) is composed of four processes: process A, process B-α, process B-β, and process B-γ. Note that process B-α, process B-β, and process B-γ will be collectively referred to as "process B" as appropriate.

[0024] Process A (the process of calculating interactions in a virtual space executed by it) receives operation signals from an unspecified number of clients (e.g., client α, client β, and client γ), places avatars corresponding to each client in the virtual space, generates viewpoints according to each position (however, the relationship between the avatar and the viewpoint does not have to be 1:1. For example, multiple clients may share one viewpoint), and calculates the operation of the avatars and real-time interactions in the virtual space.

[0025] Process A (the CG generation / visualization processes executed by it ("CG generation / visualization 1" and "CG generation / visualization 2" in the diagram)) prepares information about the sounds that occur within the virtual space due to the interactions mentioned above (such as the generation of sound sources or changes in positional relationships; sound does not have to be generated, but common sounds such as background music can be overlaid on the images). It also generates CG according to the viewpoint of each avatar (for example, two, CG1 and CG2), visualizes the CG (for example, generates "image 1" and "image 2"), and distributes the images to the client. Information about the sounds, along with the positional information of each avatar, is transmitted in real time to process B, which will be described later (reference information such as memory addresses, or data transfer such as inter-process communication or IP communication).

[0026] An SFU (a type of client-server system) may be used to distribute multiple video and audio streams to an unspecified number of clients, and to facilitate management of connections between distribution system 1 (process B type) and clients. For example, as shown in Fig. 2, video distribution from process A to clients goes through an SFU, and some video distribution is distributed by the SFU. This allows, for example, video 1 to be distributed to client α and client β (one-to-many distribution), and video 2 to be distributed to client γ (one-to-one distribution).

[0027] Process B is prepared according to the number of clients, and (process B's audio processing) generates audio according to the avatar position from the information about the sound in process A, and streams the audio to the client as compressed information. Process B generates audio from a sound source and distributes it. For example, process B-α generates audio 1 (audio according to the position of client α) and streams the generated audio 1 to client α. For another example, process B-β generates audio 2 (audio according to the position of client β) and streams the generated audio 2 to client β. For another example, process B-γ generates audio 3 (audio according to the position of client γ) and streams the generated audio 3 to client γ. The distribution of audio from process B to the client is P2P (peer-to-peer), but may be via an SFU.

[0028] Here is a concrete example of process B generating voice according to the avatar position from sound information. The generated voice is audio data that takes into account sound effects according to the position of the avatar operated by each client. For example, if user X and user Y are standing side by side on the left and right, and the sound of a bonfire is playing in the middle (information about this sound is transmitted from process A to process B. However, this is not the sound itself; the sound itself is generated in process B along with other sounds, taking into account sound effects), the voice of user X will be generated with the sound of the bonfire coming from the right, and the voice of user Y will be generated with the sound of the bonfire coming from the left.

[0029] CG generation refers to the appearance of the 3DCG at that point in time, which is done sequentially in real time (and the memory area on the server where that data is stored). Visualization refers to compressing that sequential time-series information using so-called video encoding, and making it possible to transmit it externally.

[0030] Process B holds the sound source data for generating sound, but does not perform the processing performed by process A, such as reading CG data, rendering processing, visualization, event processing within the virtual space, and physical simulation processing. Process B may use a program that can compress and distribute video with a lower processing load than the game engine.

[0031] For example, process B-α distributes to client α the sound generated in the virtual space according to the position of α. Operation signals from client α are received by process A, not process B-α.

[0032] Process A reads the sound assets in Figure 2, and common background music can be added to the video. Process B also reads the sound assets and generates audio that matches the position from the sound data in the game engine.

[0033] In Figure 2, the high-load processing required for audio generation can be performed on the video side and synchronized with the audio processing side. In other words, process A processes multiple clients and then distributes the results to process B, so it does as much processing as possible efficiently. One example of high-load processing is interactions that occur in space in response to operations by multiple clients (physics simulation, multiplayer processing).

[0034] This concludes the explanation of Process Type B. Next, we will explain the details of Process Type C.

[0035] [Process C type] The process C type of distribution system 1 executes multiple processes for processing at least one of video and audio, and executes, in separate processes, a common process (executed in process A described below) for generating video corresponding to each of multiple positions in three-dimensional space and distributing it to each client, and a distribution process (executed in process C described below) for distributing sound source information related to a single sound source, including position information of the single sound source in three-dimensional space, to each client.

[0036] The three-dimensional space may be a virtual space. The process for processing audio may only be capable of processing one line of audio. The common process and a plurality of distribution processes for a plurality of sound sources may be executed by different processes. The common process may distribute the generated video on a one-to-many basis. The common process may distribute the generated video via a Selective Forwarding Unit (SFU). The number of processes for executing the distribution process may be based on the number of sound sources. For example, the number of processes for executing the distribution process may be the same as the number of sound sources, may be equal to or greater than the number of sound sources, or may be dynamically increased or decreased according to the number of sound sources. The distribution process may not involve processing related to video. The distribution process may distribute sound source information on a one-to-many basis. The distribution process may distribute sound source information via an SFU.

[0037] Fig. 3 is a diagram showing an example of a system configuration of the distribution system 1 (process C type). In the following description of Fig. 3, description of the same contents as in Fig. 1 will be omitted as appropriate.

[0038] The distribution system 1 (process C type) is composed of multiple game engine processes that operate independently on one or multiple servers. Specifically, the distribution system 1 (process C type) is composed of three processes: process A, process C-1, and process C-2. Process C-1 and process C-2 will be collectively referred to as "process C" as appropriate.

[0039] Process A (the process of calculating interactions in a virtual space executed by it) receives operation signals from an unspecified number of clients (e.g., client α, client β, and client γ), places avatars corresponding to each client in the virtual space, generates viewpoints according to each position (however, the relationship between the avatar and the viewpoint does not have to be 1:1. For example, multiple clients may share one viewpoint), and calculates the operation of the avatars and real-time interactions in the virtual space.

[0040] Here, the positional relationship between each avatar and each sound source described in the embodiment will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the positional relationship in a virtual space (metaverse space). As shown in FIG. 4, within the virtual space indicated by a rectangle, avatar α, which is the avatar of client α, avatar β, which is the avatar of client β, avatar γ, which is the avatar of client γ, sound source 1, and sound source 2 are located. The dotted lines surrounding each sound source indicate the range within which the sound of that sound source reaches. In other words, the sound of sound source 1 reaches only avatar β, and the sound of sound source 2 reaches avatar β and avatar γ.

[0041] Process A (the CG generation and visualization process (CG generation and visualization 1 and CG generation and visualization 2 in the diagram)) prepares information about the sounds in the virtual space that are generated by the interactions described above (such as the generation of sound sources or changes in positional relationships. Sound does not have to be generated, but common sounds such as BGM can be overlaid on the video). It also generates CGs (for example, two, CG1 and CG2) according to the viewpoints of each avatar, visualizes the CGs (for example, generates "video 1" and "video 2"), and distributes the video to the client. Information about the sounds may be distributed to the client in real time together with the video distribution along with the position information of each avatar, or it may be transmitted to process C described later (reference information such as memory addresses, or data transfer such as inter-process communication or IP communication). Process A distributes the background sound common to the entire system (ambient sound, BGM) and the position information of the avatar of each client (or the client to which the sound is distributed) to the client.

[0042] CG generation refers to the appearance of the 3DCG at that point in time, which is done sequentially in real time (and the memory area on the server where that data is stored). Visualization refers to compressing that sequential time-series information using so-called video encoding, and making it possible to transmit it externally.

[0043] An SFU (a type of client-server system) may be used to distribute multiple video and audio streams to an unspecified number of clients, and to facilitate management of connections between distribution system 1 (process C type) and clients. For example, as shown in Fig. 3, video distribution from process A to clients goes through an SFU, and some video distribution is distributed by the SFU. This allows, for example, video 1 to be distributed to client α and client β (one-to-many distribution), and video 2 to be distributed to client γ (one-to-one distribution).

[0044] Process C targets one of the multiple sound sources and is prepared according to the number of sound sources. Process C distributes sound source information related to the one target sound source, including position information (sound source position information) of the one target sound source in three-dimensional space, to each client. Each client generates and plays back audio that has been acoustically processed based on the positional relationship between the sound source and the user. The distribution of sound source information from process C to the clients is via an SFU, and some of the sound source information is distributed by the SFU. As a result, for example, as shown in FIG. 4, the sound of sound source 1 has reached avatar β, and the sound of sound source 2 has reached avatar β and avatar γ, so that sound source information related to sound source 1 is distributed to client β (one-to-one distribution), and sound source information related to sound source 2 is distributed to client β and client γ (one-to-many distribution).

[0045] If the audio processing in distribution system 1 (process C type) can be streamed separately from the video, there is no need to use the same high-performance game engine as that used for the video as process C. A process that only distributes audio and has a small processing load may be used.

[0046] In distribution system 1 (process C type), a group of processes C are prepared that distribute audio in accordance with the position of the sound source in a manner independent of in-game processing, and the client receives the audio in accordance with the usage settings of the sound source (the distance between the sound source and the character controlled by the client, and in the case of dialogue, permission to make a call, etc.), and three-dimensional sound effects can be realized by performing audio processing on the client.

[0047] Reception according to usage settings can be, for example, adjusting the volume balance of background sound (BGM) and three-dimensional sound from sound source objects to change the balance between the atmosphere of the space and the sense of realism. If user X blocks (rejects communication with) user Y, and user Y starts talking to user X next to him, user X's experience will be that he can hear three-dimensional sound from other sound sources, but cannot hear what user Y is saying (because he does not want to hear it). The volume can be set to zero after reception, or the audio stream of user Y's voice source may not be received at all.

[0048] Assuming two-channel stereo audio (left and right), acoustic processing on the client could involve controlling the volume of the right stereo sound for sounds coming from the right side of the user, and lowering the volume of the left stereo sound. It could also have the sound reverberation characteristics of a space as a parameter, allowing it to be performed in a cave or bathtub, making the space reverberate well. If the client holds information about spatial walls as related information, it could also be possible to add a muffled effect, making sounds on the other side of the wall quieter.

[0049] However, sound sources in each process C may be changed in conjunction with events in the virtual space, and sound effects may be realized before distribution, separate from client processing. Also, the sound source referenced by the client may be switched, and parameters of sound effects on the client may be changed in response to event processing.

[0050] Among the information distributed from the distribution system 1 (process C type) to the client, background music (BGM) and the sound of the sound source object may be distributed to the client as an audio stream. In addition, the position information of the client and the position information of the sound source object may be distributed to the client as text information.

[0051] The above is a description of the process type C. In the following, a description will be given of the distribution system 1 which selectively uses the process type B and the process type C.

[0052] The distribution system 1 executes, as separate processes, a first distribution process for generating audio corresponding to one of a plurality of positions in three-dimensional space and distributing the audio to a single client, and a second distribution process for distributing sound source information relating to a single sound source, the sound source information including position information of the single sound source in three-dimensional space, to each client, and changes the ratio between the process for executing the first distribution process and the process for executing the second distribution process based on a predetermined criterion.

[0053] The ratio may be changed based on at least one of the number of clients or the number of sound sources. When the number of clients is less than a predetermined threshold, the ratio of processes that execute the first distribution process may be increased, or when the number of clients is equal to or greater than the predetermined threshold, the ratio of processes that execute the second distribution process may be increased. When the number of sound sources is equal to or greater than a predetermined threshold, the ratio of processes that execute the first distribution process may be increased, or when the number of sound sources is less than the predetermined threshold, the ratio of processes that execute the second distribution process may be increased. When the number of clients is less than the number of sound sources, the ratio of processes that execute the first distribution process may be increased, or when the number of clients is equal to or greater than the number of sound sources, the ratio of processes that execute the second distribution process may be increased.

[0054] The advantages and disadvantages of process type B and process type C (such as communication volume, load on distribution system 1, and client load) vary depending on the balance between the number of clients (users) and the number of sound sources, etc. Distribution system 1 switches between process type B and process type C or changes the ratio depending on the situation.

[0055] For example, when the number of clients is smaller (than a predetermined standard) and the number of sound sources is larger (than a predetermined standard), the distribution system 1 switches to process type B or increases the proportion of process type B. On the other hand, when the number of clients is larger (than a predetermined standard) and the number of sound sources is smaller (than a predetermined standard), the distribution system 1 switches to process type C or increases the proportion of process type C.

[0056] Fig. 5 is a flowchart showing an example of a ratio change process executed by the distribution system 1. As shown in Fig. 5, the distribution system 1 compares the number of clients with the number of sound sources (step S1), and if the number of clients is less than the number of sound sources (S1: YES), it may switch to process type B (step S2). On the other hand, if the number of clients is equal to or greater than the number of sound sources (S1: NO), it may switch to process type C (step S3).

[0057] Process type B and process type C are not exclusive and can be used simultaneously. This has the advantage that it is easier to perform voice processing independent of the game engine's voice processing (such as dialogue control for an unspecified number of people) in parallel.

[0058] For example, if the maximum number of sound sources is set to "60" and the number of clients (number of simultaneous users) in one space is set to "50" due to constraints on the processing performance of the distribution system 1 or SFU, and the number of clients is set to "30", then 30 process Cs handling high priority sound sources (or sound sources located close to the position of the character operated by the client, voices being spoken in an unspecified number of conversations, etc.) will be started and distributed. Figure 6 is a diagram showing an example of the system configuration of the distribution system 1 before the ratio is changed.

[0059] When the number of clients reaches "40", "10" of the running processes C with low priority (clients far away, muted, etc.) are stopped. Figure 7 is a diagram showing an example of the system configuration after changing the ratio of the distribution system 1. The distribution system 1 may stop processes C in proportion to the increase in clients, and keep the number of audio streams below a certain level.

[0060] If we consider that Process B is linked one-to-one to a client, the minimum number of sound sources that can be secured for Process C is "10". This allows for the expression of liveliness in human communication, but the fewer the number of sound sources, the more difficult it becomes to express a realistic space. Conversely, if the number of connected clients is small, more sound sources will be placed and used in the space, resulting in a more realistic sound experience.

[0061] Next, the effects of the distribution system 1 according to the embodiment will be described.

[0062] The delivery system 1 is a delivery system that executes a first delivery process, which is a sound to be delivered to one client, generating a sound corresponding to one of a plurality of positions in a three-dimensional space and delivering it to the one client, and a second delivery process, which is sound source information related to a sound source, including position information of the one sound source in the three-dimensional space, to each client, in different processes, and changes the ratio of the process that executes the first delivery process to the process that executes the second delivery process based on a predetermined criterion. With this configuration, in the process in which the first delivery process is executed, a sound corresponding to one position in the three-dimensional space is generated and delivered to the one client. Therefore, it is possible to deliver an appropriate sound to the client. Also, in the process in which the second delivery process is executed, sound source information related to the one sound source, including position information of the one sound source in the three-dimensional space, is delivered to each client. For example, each client can play an appropriate sound based on the delivered sound source information. Therefore, it is possible to deliver an appropriate sound to the client. In addition, since the ratio between the processes executing the first distribution process and the processes executing the second distribution process is changed, the ratio can be appropriately changed depending on, for example, the situation (communication volume, load on the distribution system 1, client load, etc.).

[0063] Furthermore, the ratio may be changed based on at least one of the number of clients or the number of sound sources in the distribution system 1. With this configuration, the ratio can be appropriately changed according to the situation (such as the amount of communication, the load on the distribution system 1, or the client load) based on at least one of the number of clients or the number of sound sources, for example.

[0064] In addition, in the distribution system 1, when the number of clients is less than a predetermined threshold, the ratio of processes that execute the first distribution process may be increased, or when the number of clients is equal to or greater than the predetermined threshold, the ratio may be increased. With this configuration, the ratio can be appropriately changed according to, for example, a situation based on the number of clients (such as communication volume, load on the distribution system 1, and client load).

[0065] In addition, in the distribution system 1, when the number of sound sources is equal to or greater than a predetermined threshold, the ratio of processes that execute the first distribution process may be increased, or when the number of sound sources is less than the predetermined threshold, the ratio of processes that execute the second distribution process may be increased. With this configuration, the ratio can be appropriately changed according to, for example, a situation based on the number of sound sources (such as communication volume, load on the distribution system 1, and client load).

[0066] In addition, in the distribution system 1, when the number of clients is less than the number of sound sources, the ratio of processes that execute the first distribution process may be increased, or when the number of clients is equal to or greater than the number of sound sources, the ratio may be increased. With this configuration, the ratio can be appropriately changed according to the situation based on the relationship between the number of clients and the number of sound sources (such as communication volume, load on the distribution system 1, and client load).

[0067] According to distribution system 1, by using process A which performs common and complex processing and process B which generates and distributes audio according to each client, it is possible to process virtual spaces using multiple systems of video and audio in a single system (or server) with a minimal increase in processing load (compared to multiple game engines (processes) which individually perform all virtual space processing for each client and synchronization processing between game engines).

[0068] According to delivery system 1, video and audio are delivered separately by delivery system 1, and the video and audio are played back simultaneously on the client for use. Because the only individual information is audio, the GPU (Graphics Processing Unit) server load and communication volume for video delivery can be minimized. This enables a system configuration that can deliver multiple content and reduce communication costs. In delivery system 1, audio is generated as it sounds from individual user positions, and the audio at the user's position is received on the user (client) side, and is played back on the client together with the separately delivered video.

[0069] According to the distribution system 1, even if one game engine can only process one sound, it is possible to prepare and distribute the sound corresponding to each of multiple clients. Also, according to the distribution system 1, the video processing for generating the video does not depend on the number of clients, so the load on the distribution system 1 can be reduced. Also, according to the distribution system 1, the processing for synthesizing the video and the sound is performed on the client side, so the load on the distribution system 1 can be reduced.

[0070] According to distribution system 1, by using process A which performs common and complex processing and process C which distributes sound source information according to each client, it is possible to process virtual spaces using multiple lines of video and audio in a single system (or server) with a minimal increase in processing load (compared to a case where multiple game engines (processes) perform all virtual space processing corresponding to each client and synchronization processing between game engines individually).

[0071] In distribution system 1, the sound source itself is distributed in addition to the video, and the video and sound source are played back simultaneously on the client. This minimizes the load on distribution system 1. In distribution system 1, background sound (ambient sound) common to the entire system and the sound and position of each sound source object are streamed separately, and the sound is processed on the client side according to the client's avatar position and the sound is played back. Note that background music and the like are streamed together with the video, and only the sound of each individual sound source object is streamed.

[0072] According to the distribution system 1, even if one game engine can process only one sound, each of multiple clients can play back video and audio according to the client. According to the distribution system 1, the video processing for generating video does not depend on the number of clients, so the load on the distribution system 1 can be reduced. According to the distribution system 1, the process for executing the distribution processing does not depend on the number of clients, so the load on the distribution system 1 can be reduced. According to the distribution system 1, the process for generating audio is performed on the client side, so the load on the distribution system 1 can be reduced. According to the distribution system 1, the process for synthesizing video and audio is performed on the client side, so the load on the distribution system 1 can be reduced.

[0073] According to the distribution system 1, it is possible to switch to a distribution method that takes into consideration the communication volume, server load, and client load depending on the number of clients and the number of sound sources.

[0074] A conventional problem is that general game engines are designed to output only one audio channel. Therefore, even if multiple camera viewpoints are streamed on one game engine, the audio will not be tailored to the camera viewpoints, and will only be a single channel common to the entire game, making it impossible to stream multiple audio streams tailored to each avatar position (multiple camera viewpoints). In addition, even if one rendered video is distributed to multiple clients using cloud rendering, audio tailored to the avatar position cannot be achieved.

[0075] The distribution system 1 solves the above problem by, for example, executing multiple game engine processes in a single cloud rendering environment, and performing and distributing audio processing based on location information in additional game engine processes among the multiple game engine processes.

[0076] The delivery system 1 realizes a spatial audio control method using cloud rendering. The delivery system 1 realizes a spatial audio control method based on the ratio of the number of viewers to the number of sound sources.

[0077] The distribution system 1 of the present disclosure may have the following configuration.

[0078] [1] A distribution system that executes, in separate processes, a first distribution process for generating a sound corresponding to one of a plurality of positions in a three-dimensional space to be distributed to one client, and distributing the sound to the one client, and a second distribution process for distributing sound source information related to a sound source, the sound source information including position information of the one sound source in the three-dimensional space, to each client, changing a ratio of the process that executes the first distribution process to the process that executes the second distribution process based on a predetermined criterion; Delivery system.

[0079] [2] changing the ratio based on at least one of the number of clients or the number of sound sources; The distribution system according to [1].

[0080] [3] The number of clients is If the number of processes is less than a predetermined threshold, increasing the proportion of processes that execute the first distribution process; or When the number of processes is equal to or greater than a predetermined threshold, the ratio of processes that execute the second delivery process is increased. A distribution system according to [1] or [2].

[0081] [4] The number of the sound sources is When the ratio is equal to or greater than a predetermined threshold, the ratio of processes that execute the first distribution process is increased; or When the number of processes is less than a predetermined threshold, the ratio of processes that execute the second distribution process is increased. A distribution system according to any one of [1] to [3].

[0082] [5] When the number of the clients is less than the number of the sound sources, increasing the proportion of processes that execute the first distribution process; or When the number of the clients is equal to or greater than the number of the sound sources, a ratio of processes that execute the second distribution process is increased. A distribution system according to any one of [1] to [4].

[0083] The processes used in the description of the above embodiments are realized by any combination of at least one of hardware and software. The method of realizing the processes is not particularly limited. That is, the processes may be realized by using one device that is physically or logically coupled, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wirelessly, etc.). The processes may be realized by combining the one device or the multiple devices with software.

[0084] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, assignment, etc. For example, a functional block (component) that performs the function of transmission is called a transmitting unit or a transmitter. In either case, as described above, there is no particular limitation on the method of realization.

[0085] For example, the distribution system 1 according to an embodiment of the present disclosure may function as a computer that performs processing of the method of the present disclosure. Fig. 8 is a diagram showing an example of a hardware configuration of the distribution system 1 according to an embodiment of the present disclosure. The above-mentioned distribution system 1 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

[0086] In the following description, the term "apparatus" may be replaced with a circuit, a device, a unit, etc. The hardware configuration of the distribution system 1 may be configured to include one or more of the apparatuses shown in the drawings, or may be configured to exclude some of the apparatuses.

[0087] Each function in the distribution system 1 is realized by loading a specific software (program) onto hardware such as a processor 1001 and a memory 1002, so that the processor 1001 performs calculations, controls communications by a communication device 1004, and controls at least one of reading and writing of data in the memory 1002 and the storage 1003.

[0088] The processor 1001 controls the entire computer by running, for example, an operating system, and may be configured with a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, and the like.

[0089] Furthermore, the processor 1001 reads out programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to these. As the programs, those that cause a computer to execute at least a part of the operations described in the above-mentioned embodiments are used. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The programs may be transmitted from a network via a telecommunications line.

[0090] The memory 1002 is a computer-readable recording medium, and may be configured by at least one of, for example, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Random Access Memory (RAM), etc. The memory 1002 may be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store a program (program code), software modules, etc. that are executable to implement a wireless communication method according to an embodiment of the present disclosure.

[0091] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy (registered trademark) disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of memory 1002 and storage 1003.

[0092] The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 1004 may be configured to include a high-frequency switch, a duplexer, a filter, a frequency synthesizer, etc., in order to realize, for example, at least one of Frequency Division Duplex (FDD) and Time Division Duplex (TDD).

[0093] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0094] In addition, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between each device.

[0095] Furthermore, the distribution system 1 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these pieces of hardware.

[0096] Notification of information is not limited to the aspects / embodiments described in this disclosure and may be performed using other methods.

[0097] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0098] The input and output information may be stored in a specific location (e.g., memory) or may be managed using a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.

[0099] The determination may be based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0100] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

[0101] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0102] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0103] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0104] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0105] In addition, terms explained in this disclosure and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings.

[0106] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0107] Furthermore, the information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information.

[0108] The names used for the above parameters are not limiting in any way, and furthermore, the formulas etc. using these parameters may differ from those explicitly disclosed in this disclosure.

[0109] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.

[0110] The terms "connected," "coupled," or any variation thereof, refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be physical, logical, or a combination thereof. For example, "connected" may be read as "access." As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and optical (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0111] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0112] Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0113] The "means" in the configuration of each of the above devices may be replaced with "part," "circuit," "device," etc.

[0114] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Further, when used in this disclosure, the term "or" is not intended to be an exclusive or.

[0115] In this disclosure, where articles have been added due to translation, such as, for example, a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0116] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different." [Explanation of symbols]

[0117] 1...distribution system, 1001...processor, 1002...memory, 1003...storage, 1004...communication device, 1005...input device, 1006...output device, 1007...bus.

Claims

1. A distribution system that executes, in separate processes, a first distribution process for generating a sound corresponding to one of a plurality of positions in a three-dimensional space to be distributed to one client, and distributing the sound to the one client, and a second distribution process for distributing sound source information related to a sound source, the sound source information including position information of the one sound source in the three-dimensional space, to each client, changing a ratio of the process that executes the first distribution process to the process that executes the second distribution process based on a predetermined criterion; Delivery system.

2. changing the ratio based on at least one of the number of clients or the number of sound sources; The distribution system according to claim 1 .

3. The number of clients is If the number of processes is less than a predetermined threshold, increasing the proportion of processes that execute the first distribution process; or When the number of processes is equal to or greater than a predetermined threshold, the ratio of processes that execute the second delivery process is increased. The distribution system according to claim 1 .

4. The number of the sound sources is When the ratio is equal to or greater than a predetermined threshold, the ratio of processes that execute the first distribution process is increased; or When the number of processes is less than a predetermined threshold, the ratio of processes that execute the second distribution process is increased. The distribution system according to claim 1 .

5. When the number of the clients is less than the number of the sound sources, increasing the proportion of processes that execute the first distribution process; or When the number of the clients is equal to or greater than the number of the sound sources, a ratio of processes that execute the second distribution process is increased. The distribution system according to claim 1 .

Images