Apparatus, method, and computer program for selecting a mode for an audio stream input format.

Abstract

An example of the present disclosure relates to selecting a mode for the input format of an audio stream containing signals mixed from different sources. An indication of one or more modes available for the selected input format of a first audio signal is obtained. An indication of one or more modes available for the selected input format of a second audio signal is also obtained. The second audio signal is combined to form an audio stream. In the example, a mode is selected for an audio stream comprising the first audio signal and the second audio signal, and the selection is at least in part based on one or more common modes available for the selected input format of the first audio signal and the selected input format of the second audio signal.

Description

Technical Field

[0001] Examples of the present disclosure relate to an apparatus, a method, and a computer program for selecting a mode for an input format of an audio stream. Some relate to an apparatus, a method, and a computer program for selecting a mode for an input format of an audio stream that includes signals mixed from different sources.

Background Art

[0002] Audio applications such as teleconferences can acquire audio signals from different sources or capture setups. These different signals can be mixed to generate an audio stream that can be transmitted to participants in a teleconference or other audio application. If signals from different sources or capture setups use different modes for an input, this can negatively affect the quality of the signal components within the audio stream.

Summary of the Invention

[0003] In some but not all, various examples of the present disclosure provide an apparatus comprising: means for obtaining instructions for one or more modes available for a selected input format of a first audio signal, wherein the first audio signal is received from a first source; means for obtaining instructions for one or more modes available for a selected input format of a second audio signal, wherein the second audio signal is received from a second source, and the first audio signal and the second audio signal are combined to form an audio stream; means for selecting a mode for an audio stream comprising the first audio signal and the second audio signal, wherein the selecting means is at least partially based on one or more common modes available for the selected input format of the first audio signal and the selected input format of the second audio signal; and means for transmitting instructions for the selected mode to the respective sources.

[0004] The indication of the available modes may be received for two or more audio signals from different sources, the different audio signals may have different modes available for the selected input format, and the audio signals are combined to form an audio stream.

[0005] The selected mode may include a mode common to two or more audio signals from two or more sources.

[0006] The mode may be selected based in part on at least one of the following: the mode most commonly available for the source and the selected input format; the computational load for mixing the audio signals from the source into a combined stream in each of the available modes for the selected input format; and the computational load for decoding the audio signals from the source and / or encoding the mixed audio stream in each of the available modes for the selected input format.

[0007] The means may include, in the selected mode, means for mixing at least the first audio signal and at least the second audio signal using a common mode to generate a mixed audio stream, and means for enabling the transmission of the mixed audio stream.

[0008] The selected mode may include a first mode for the audio mainstream and a second mode for the audio substream.

[0009] The mainstream may include mixing two or more audio signals having a common mode in the selected input format.

[0010] The means described above may be means that enable the transmission of the main stream and the substream without mixing the main stream and the substream.

[0011] The means may include means for receiving instructions for a preferred mode for an end-user device, and means for using the instructions for the preferred mode to assist in selecting the mode for the audio stream.

[0012] The aforementioned audio signal may include a metadata-assisted spatial audio signal.

[0013] The selected input format may include metadata-assisted spatial audio formats.

[0014] The one or more modes available for the input format may have time-frequency resolution.

[0015] At least one of the available modes may have higher frequency resolution, and at least one of the available modes may have higher temporal resolution.

[0016] The means may be means for generating spatial metadata using the selected mode.

[0017] Various examples of the present disclosure, though not necessarily all, provide a method comprising: obtaining instructions for one or more modes available for a selected input format of a first audio signal, the first audio signal being received from a first source; obtaining instructions for one or more modes available for a selected input format of a second audio signal, the second audio signal being received from a second source, the first audio signal and the second audio signal being combined to form an audio stream; selecting a mode for the audio stream comprising the first audio signal and the second audio signal, the selecting step being at least in part based on one or more common modes available for the selected input format of the first audio signal and the selected input format of the second audio signal; and transmitting instructions for the selected mode to the respective sources.

[0018] By various examples of the present disclosure, though not necessarily all, there is a computer program that includes program instructions, which, when executed by a device, causes the device to perform at least the steps of: obtaining instructions for one or more modes available for a selected input format of a first audio signal, wherein the first audio signal is received from a first source; obtaining instructions for one or more modes available for a selected input format of a second audio signal, wherein the second audio signal is received from a second source, and the first audio signal and the second audio signal are combined to form an audio stream; selecting a mode for the audio stream, comprising the first audio signal and the second audio signal, wherein the selecting step is at least partially based on one or more common modes available for the selected input format of the first audio signal and the selected input format of the second audio signal; and transmitting instructions for the selected mode to the respective sources.

[0019] Although the above examples and optional features of this disclosure are described separately, it should be understood that their provision in all possible combinations and permutations is included within this disclosure. It should be understood that various examples of this disclosure may include any or all of the features described with respect to other examples of this disclosure, and vice versa. It should also be understood that any one or more or all of the features may be implemented, executed, and made executable by apparatus, methods, and / or computer program instructions, in any combination, as desired and as necessary.

[0020] Refer to the attached diagrams to illustrate some examples. [Brief explanation of the drawing]

[0021] [Figure 1A] It is a diagram showing an exemplary system. [Figure 1B] It is a diagram showing an exemplary system. [Figure 2] It is a diagram showing an exemplary metadata frame. [Figure 3A] It is a diagram showing an exemplary metadata frame. [Figure 3B] It is a diagram showing an exemplary metadata frame. [Figure 4] It is a diagram showing the mixing of an audio stream. [Figure 5] It is a diagram showing an exemplary method. [Figure 6] It is a diagram showing the mixing of an audio stream. [Figure 7] It is a diagram showing the mixing of an audio stream. [Figure 8] It is a diagram showing an exemplary method. [Figure 9] It is a diagram showing an exemplary system. [Figure 10] It is a diagram showing an exemplary device.

[0022] The drawings are not necessarily to scale. Certain features and diagrams in the drawings may be shown schematically or with exaggerated scale for clarity and brevity. For example, the dimensions of some elements in the figures may be exaggerated relative to other elements to aid in clarification. Corresponding reference numerals are used in the figures to indicate corresponding features. For ease of understanding, not all reference numerals are necessarily shown in all figures.

Mode for Carrying Out the Invention

[0023] Examples of the present disclosure may be implemented in an audio system that utilizes an immersive voice audio service (IVAS) codec.

[0024] Figures 1A and 1B schematically illustrate an exemplary system 101 that may be used to implement an example of the present disclosure. The exemplary system may use an IVAS codec.

[0025] System 101 is configured to allow audio to be captured by participant devices 105 at one or more locations and transmitted to other participant devices 105 within System 101. This allows audio to be shared between different participant devices 105 within System 101. System 101 can be used for remote conferencing or any other suitable audio application.

[0026] In the example shown in Figure 1A, system 101 comprises a control unit 103 and three participant devices 105A, 105B, and 105C. In other examples, system 101 may comprise other types and numbers of devices.

[0027] The control unit 103 is configured to receive audio signals from sources within the system 101. In this example, the sources may be participant devices 105A, 105B, and 105C. The audio signals from participant devices 105A, 105B, and 105C may include audio generated by the user 111 of participant device 105, such as a voice signal or any other suitable type of audio.

[0028] Participant devices 105A, 105B, and 105C may comprise any suitable type of device. Participant devices 105A, 105B, and 105C may comprise any other suitable type of device that can be configured to capture audio and provide the audio signal to user 111.

[0029] System 101 is configured so that participant devices 105A, 105B, and 105C transmit upstream signals 107 to the control unit 103. Upstream signals 107 may include audio captured by each participant device 105A, 105B, and 105C. The audio can come from any sound source at the location of each participant device 105A, 105B, and 105C. In the example in Figure 1A, the sound source includes user 111 using participant devices 105A, 105B, and 105C. User 111 may be having a conversation or communicating in another way during the remote conference. Therefore, different participant devices 105A, 105B, and 105C provide different sources of audio signals.

[0030] The control unit 103 can be configured to receive audio signals from multiple sources. In this case, the audio signals include upstream signals 107A, 107B, and 107C from participant devices 105A, 105B, and 105C.

[0031] The control unit is configured to mix audio signals from multiple sources to generate an audio stream. The control unit 103 can then provide the audio streams of the downstream signals 109A, 109B, and 109C to the respective participant devices 105A, 105B, and 105C. This allows the audio content to be shared among multiple different participant devices 105A, 105B, and 105C located in different locations.

[0032] System 101 is configured such that a first participant device 105A receives a downstream signal 109A from a control unit 103. The downstream signal 109A includes packets based on an input format. The input format may be Metadata-Assisted Spatial Audio (MASA) or any other suitable format. The downstream signal 109A transmitted from the control unit 103 to the first participant device 105A is formed based on input audio signals from other participant devices 105B, 105C within System 101. The control unit 103 is configured to decode the upstream signals received from the other participant devices 105B, 105C and mix the decoded signals into a combined format. The mixed signals can then be encoded for transmission to the first participant device 105A. The first participant device 105A receives the packets in the downstream signal 109A, decodes the IVAS bitstream in the signal, and renders the signal using a suitable format for playback to user 111 of the first participant device 105A. For example, the signal can be rendered binaurally for playback via headphones. The control unit can also mix and transmit signals for other participant devices 105B, 105C.

[0033] Figure 1B shows a different system 101 without the control unit 103. In this system 101, audio signals can be sent directly between participant devices 105A, 105B, and 105C.

[0034] In the system shown in Figure 1, the first audio signal 113A is transmitted from the first participant device 105A to the second participant device 105B, the second audio signal 113B is transmitted from the second participant device 105B to the second participant device 105A, the third audio signal 113C is transmitted from the first participant device 105A to the third participant device 105C, the fourth audio signal 113D is transmitted from the third participant device 105C to the first participant device 105A, the fifth audio signal 113E is transmitted from the second participant device 105B to the third participant device 105C, and the sixth audio signal 113E is transmitted from the third participant device 105C to the second participant device 105B.

[0035] The audio signals transmitted by participant devices 105A, 105B, and 105C may include audio captured by each participant device 105A, 105B, and 105C. The audio signals captured by the participant devices may also be mixed with audio signals received from other sources, such as other participant devices 105A, 105B, and 105C.

[0036] As shown in both Figure 1A and Figure 1B, multiple users 111 may be using participant devices 105A, 105B, and 105C. For example, in the examples in Figures 1A and 1B, three users 111 are using the first participant device 105A, only one user 111 is using the second participant device 105B, and only one user 111 is using the third participant device 105C. In other examples, system 101 may include a different number of users 111 and participant devices 105.

[0037] The IVAS codec can be used in telecommunications systems 101, such as system 101 in Figures 1A and 1B. The IVAS codec can support various input formats. These various input formats can include stereo, multi-channel (MC), object-based audio (ISM), scene-based audio (SBA), and MASA. Several combinations can be supported by various means, for example, by using combined formats such as objects with MASA (OMASA) or objects with SBA (OSBA). There are also separate input formats for binaural audio that can operate in the same way as stereo inputs.

[0038] The MASA input format uses one or more audio signals along with corresponding spatial metadata. MASA spatial metadata parameters describe the spatial characteristics of the captured spatial sound scene. Spatial metadata can include information such as direction and direct-to-total energy ratio in the frequency band, or any other relevant spatial information.

[0039] The MASA audio stream can be acquired by a participant device 105 within System 101, such as System 101 shown in Figures 1A and 1B. For example, the MASA audio stream can be acquired by capturing spatial audio with the microphone of the participant device 105, and the corresponding spatial metadata can be estimated based on the microphone signal.

[0040] MASA streams can also be obtained from other sources, such as specific spatial audio microphones (like ambisonics), studio mixes (like 5.1 mixes), or other content, by performing appropriate format conversions.

[0041] It is also possible to use the MASA tool within the codec to encode a multichannel signal by converting the multichannel signal to a MASA stream and then encoding that stream.

[0042] This parametric representation of MASA metadata is based on frequency bands. Certain spatial characteristics are related to frequency bands, and adjacent frequency bands can exhibit different characteristics. In the case of the MASA format, 24 frequency bands are used. A metadata frame for a 20ms audio frame is divided into four subframes, each 5ms long. Therefore, the parametric representation in each frame consists of 24 frequency bands across four time slots, providing a total of 96 time-frequency tiles.

[0043] The frame size in IVAS is 20 ms, and therefore the time subframe is 5 ms. Furthermore, MASA supports one or two directions for each time-frequency tile (i.e., for each time-frequency tile there are one or two directional indices, a direct-to-total energy ratio, and a diffusion coherence parameter).

[0044] Figure 2 schematically shows the time-frequency resolution of the IVAS MASA metadata frame. The metadata frame, configured to be processed by the IVAS encoder, contains 96 time-frequency (TF) tiles.

[0045] Various encoding methods, particularly at lower bitrates, can reduce the effective TF resolution. This reduction can occur in terms of reducing only the temporal resolution, only the frequency resolution, or both. The effective input TF resolution may also differ from that supported by the MASA format. For example, repeating the same parameter values ​​for time subframes 0, 1, 2, and 3 can lead to a reduction in temporal resolution. Similarly, some frequency bands may share the same parameter values ​​as several adjacent frequency bands within the same time subframe, resulting in a reduction in frequency resolution.

[0046] Figures 3A and 3B schematically illustrate the degrading of TF resolution in an exemplary IVAS MASA metadata frame. Different TF resolutions may be used in different modes used for the MASA input format.

[0047] The metadata frame in Figure 3A shows the reduced time and frequency resolution. The metadata frame in Figure 3A has 12 effective frequency bands and one effective time subframe. This provides 12 TF tiles.

[0048] The metadata frame in Figure 3B shows another frame with reduced time and frequency resolution. The metadata frame in Figure 3B has six effective frequency bands and four effective time subframes. This provides 24 TF tiles.

[0049] The metadata frames shown in Figures 3A and 3B are merely examples. Other reductions in TF resolution can be used in different modes of the MASA input format. For example, one or more input modes may use 18 frequency bands and one time subframe, which provides 18 TF tiles. Another or more input modes may use 5 frequency bands and 4 time subframes, which provides 20 TF tiles.

[0050] The metadata frames shown in Figures 3A and 3B are schematic diagrams of the frame. The input IVAS MASA metadata frame must have 96 TF tiles. The (effective) TF resolution can only be reduced by repeating parameter values. However, in other contexts, the effective TF resolution may correspond to the actual representation. For example, within a codec, only repeated values ​​are "remembered".

[0051] A decrease in bitrate may necessitate a reduction in TF resolution as part of the encoding process. In scenarios with fewer metadata parameter values ​​to encode, it is easier to encode them all with reasonable precision for playback in the decoder / renderer. This reduction can generally be done based on the content (values) of the input spatial metadata, but often it can also be done by considering transport signal characteristics such as their energy. Transport signal characteristics can also be analyzed on a per-TF tile basis.

[0052] The problem may arise in system 101 where audio from different sources is mixed. If the audio sources use different modes of the input format, the audio sources may have different TF resolutions. As an exemplary example, in system 101 of Figure 1A, a second participant device 105B may generate a MASA input format with full TF resolution. An IVAS encoder on the second participant device 105B encodes the input audio signal according to the negotiated bitrate and transmits the encoded signal to the control unit 103. A third participant device 105C may generate a MASA input format with reduced TF resolution. An IVAS encoder on the third participant device 105C also encodes the input audio signal according to the negotiated bitrate and transmits the encoded signal to the control unit 103. A significant reduction in TF resolution may apply to both input streams due to the bitrate. The reduction may differ for each participant device 105. For example, the encoder on the second participant device 105B is likely to be configured to maintain temporal resolution, resulting in a greater loss of frequency resolution, while the encoder on the third participant device 105C may be configured to maintain frequency resolution because the input signal already has a lower effective temporal resolution.

[0053] While the individual input signals from each participant device 105B and 105C may be of good quality, differences in TF resolution can become a problem when the different input signals are mixed by the control unit 103. If the mixing is configured to maintain the best possible quality for both input signals, the resulting mixed signal will have high temporal resolution (from the second participant device 105B) and high frequency resolution (from the third participant device 105C). However, in order to encode the mixed signal for transmission to the first participant device 105A at the available bitrate, a new TF resolution reduction may need to be performed. Often, this can degrade the quality of at least one of the input signals or component input signals of the mix. In this particular example, the input signal from the third participant device 105C may be significantly affected.

[0054] Figure 4 schematically illustrates how mixing audio streams and mixing input signals with different TF resolutions can affect the quality of the output signal.

[0055] In Figure 4, the upstream signal 107B from the second participant device 105B and the upstream signal 107C from the third participant device 105C are received by the control unit 103. The metadata frames 401 for each of the upstream signals 107B and 107C are shown in Figure 4.

[0056] The metadata frame 401B for the upstream signal 107B from the second participant device 105B exhibits reduced frequency resolution. The metadata frame 401B has three effective frequency bands and four effective time subframes. This provides 12 TF tiles.

[0057] The metadata frame 401C for the upstream signal 107C from the third participant device 105C exhibits reduced frequency resolution and reduced temporal resolution. The metadata frame 401C has six effective frequency bands and one effective temporal subframe. This provides six TF tiles.

[0058] The control unit 103 includes a decoder 403, a mixer 405, and an encoder 407.

[0059] The input signals 107B and 107C from the respective participant devices 105B and 105D are provided as inputs to the decoder 403. The decoder 403 decodes the input signals 107B and 107C and supplies the decoded input signals to the mixer 405.

[0060] Mixer 405 mixes the decoded input signals. Mixer 405 generates an intermediate mix 409, which is provided as input to encoder 407. Mixer 405 can be configured to retain as much information as possible in the intermediate mix 409. In the example in Figure 4, the intermediate mix has six effective frequency bands and four effective time subframes. This provides 24 TF tiles.

[0061] Encoder 407 is configured to mix the intermediate mix and provide a downstream signal 109A that can be transmitted to the first participant device 105A. Encoder 407 may need to reduce the TF resolution due to bitrate or other reasons.

[0062] In this case, the output of encoder 409 has three effective frequency bands and four effective time subframes. This provides 12 TF tiles. The TF resolution of signal 107B from the second participant device 105B is maintained, but the TF resolution of signal 107C from the third participant device 105C is not maintained. Signal 107C from the third participant device 105C does not acquire any time resolution in the operation performed by control unit 103, because this information cannot be added.

[0063] This reduction in TF resolution for at least some of the components of the output signal may degrade the audio quality for the user 111 of participant device 105. An example of this disclosure is configured to avoid this degradation of audio quality.

[0064] Figure 5 shows an exemplary method. This method may be implemented by a control unit and / or participant device 105 and / or any other suitable device within the exemplary system 101. This method may be performed during session negotiation or at any other suitable time.

[0065] In block 501, the method includes the step of obtaining an indication of one or more modes available for a selected input format of a first audio signal, wherein the first audio signal is received from a first source.

[0066] In block 503, the method includes the step of obtaining an instruction for one or more available modes for a selected input format of a second audio signal, wherein the second audio signal is received from a second source. The first audio signal and the second audio signal are combined to form an audio stream.

[0067] In the example in Figure 5, the available mode indication is received for two different audio signals from two different sources. The available mode indication may be received for two or more audio signals from different sources, and the different audio signals may have different modes available for the selected input format. Multiple different audio signals from multiple different sources can be combined to form an audio stream.

[0068] The input signal may be an upstream audio signal as shown in Figure 1A, or any other suitable type of signal. The source from which the audio signal is received may be the participant device 105 in system 101, or any other suitable type of source for the audio signal.

[0069] The input format specifies the format used for the signal provided to the encoder. Various input formats can include stereo, MC, ISM, SBA, and MASA, and / or any other type of format.

[0070] An input format can have different modes available. These different modes may include different operating modes for the input format. For example, the input modes available for the MASA input format may include high frequency resolution (HFR, 1 subframe, 1 sf mode) and high temporal resolution (HTR, 4 subframes, 4 sf mode).

[0071] The available modes for an input format have a TF resolution. Different modes may have different TF resolutions. In some examples, at least one of the available modes has a higher frequency resolution, and at least one of the available modes has a higher temporal resolution.

[0072] In block 505, the method includes the step of selecting a mode for an audio stream comprising a first audio signal and a second audio signal. The mode selection is at least in part based on one or more common modes available for a selected input format for the first audio signal and a selected input format for the second audio signal.

[0073] In some examples, the selected mode may include a mode common to two or more audio signals, where the audio signals come from two or more sources. For example, if there are modes available for both the input format of the first audio signal and the input format of the second audio signal, this could be the selected mode.

[0074] In some examples, the selected mode may be chosen based at least partially on the most commonly available mode for the input format selected for each source. The most commonly available mode may be the mode that is shown to be available for the input format selected for most audio signals. The most common mode may be the mode available for the maximum number of participant devices 105.

[0075] In some cases, the mode may be selected based on the audio quality it offers. If a particular mode offers better audio quality than other available modes, the mode with the better audio quality can be selected.

[0076] In some examples, the mode may be selected based on the capabilities of the control unit 103 or any other suitable device. For example, the mode may be selected based on the computational load required to mix the input audio signal into an audio stream.

[0077] In some examples, the selected mode may be chosen at least partially based on the computational load for mixing the audio signals from the source into a combined stream within the available modes for the selected input format. In such examples, a mode that results in a lower computational load for mixing may be preferred over a mode that results in a higher computational load.

[0078] In some examples, the selected mode may be chosen at least in part based on the computational load for decoding the audio signal from the source and / or encoding the mixed audio stream within the respective available modes for the selected input format. In such examples, a mode that results in a lower computational load for decoding and / or encoding may be preferred over a mode that results in a higher computational load.

[0079] In some examples, selecting a mode for an audio stream may include selecting multiple modes. In such cases, the mixed audio stream may contain different components. For example, an audio stream may consist of an audio main stream and one or more audio substreams. The main stream may differ from the substreams in that the main stream may take precedence over any substreams and / or the main stream may contain more data than one or more substreams. In such cases, the selected modes may consist of a first mode for the audio main stream and a second mode for the audio substreams.

[0080] In some examples, each component of a mixed audio stream may contain a mix of two or more audio signals. For example, the main stream and / or one or more substreams may contain a mix of two or more audio signals that share a common mode in the selected input format.

[0081] In cases where an audio stream contains different components, a mixed audio stream can be transmitted without mixing each component separately. For example, a mixed audio stream can be transmitted without mixing the main stream with one or more substreams.

[0082] In block 507, the method includes sending instructions for the selected mode to the respective sources.

[0083] In some examples, the method may include additional blocks not shown in Figure 5. For example, in some examples, the method may include the steps of mixing at least a first audio signal and at least a second audio signal using a common mode to produce an audio stream mixed in a selected mode, and enabling transmission of the mixed audio stream. The mixed audio stream can be transmitted to participant device 105 or any other suitable device, enabling the audio to be played back to the user.

[0084] In some examples, the method may include the steps of receiving a preferred mode instruction for participant device 105 and using the preferred mode instruction to assist in selecting a mode for the audio stream. Participant device 105 may be associated with an end user; that is, the participant device may be a device on which the mixed audio stream is to be transmitted.

[0085] The audio signal used may be a metadata-assisted spatial audio signal or any other suitable type of signal. The selected input format used for the audio signal may include the MASA format or any other suitable type of format. The method may also include a step of generating spatial metadata using a selected mode.

[0086] An example of this disclosure may be used to address the TF resolution problem that arises when mixing audio signals of different modes, as shown in Figure 4. In this example, participant devices 105B and 105C each indicate which modes are available for the selected input format. Different modes may have different TF resolutions.

[0087] If one or more of the participant devices 105B, 105C have only one available mode, the control unit 103 may require all participant devices 105B, 105C to use the same mode to generate the audio signal for input to the decoder 403. If the participant devices 105B, 105C have multiple available modes for the selected input format, the control unit 103 may select the mode to be used for the mixed audio stream. The mode may be selected to preserve most of the sender quality at a given bitrate.

[0088] For example, if one or more transmission devices cannot use a mode with high temporal resolution (e.g., 4sf mode), the control unit 103 can select a mode with low temporal resolution (e.g., 1sf mode). The control unit 103 can then send instructions for this selected mode with low temporal resolution to all of the relevant participant devices 105. Each participant device can then use this mode to generate an audio signal for transmission.

[0089] In some cases, the participant device 105 may need to modify the audio signal to adjust it to a selected mode. In this case, the participant device 105 needs to reduce the temporal resolution of the audio signal. The temporal resolution of the audio signal can be reduced by selecting one of the subframe values ​​(for each parameter of each frequency) and overwriting the other subframes with this value. Other modifications to reduce temporal resolution may be used in other examples. In some examples, spatial analysis with lower temporal (and potentially higher frequency) resolution can be used.

[0090] Figure 6 schematically illustrates the mixing of audio streams. In this example, the control unit 103 receives input signals 107 from three transmitting participant devices 105. The control unit 103 may be configured to select an appropriate mode for the mixed audio stream and generate a mixed audio stream so that the mixed audio stream can be transmitted to the receiving participant device 105A.

[0091] In Figure 6, the upstream signal 107B from the second participant device 105B, the upstream signal 107C from the third participant device 105C, and the upstream signal 107D from the fourth participant device 105D are received by the control unit 103. The metadata frames 401 for each of the upstream signals 107B, 107C, and 107D are shown in Figure 6. The upstream signal 107 may be in MASA format or any other suitable format.

[0092] The upstream signal 107B from the second participant device 105B has a first mode. Figure 6 shows an exemplary metadata frame 401B for the upstream signal 107B from the second participant device 105B using this first mode. This mode has reduced frequency resolution. The metadata frame 401B has three effective frequency bands and four effective time subframes. This provides 12 TF tiles.

[0093] The upstream signal 107C from the third participant device 105C has the same mode as the upstream signal 107B from the second participant device 105B. The signal 107C from the third participant device 107C and the signal 107B from the second participant device 105B have the same TF resolution. The metadata frame 401C for the upstream signal 107C from the third participant device 105C has three effective frequency bands and four effective time subframes that provide 12 TF tiles.

[0094] The upstream signal 107D from the fourth participant device 105D has a different mode from the upstream signal 107B from the second participant device 105B. The metadata frame 401D for the upstream signal 107D from the fourth participant device 105D shows reduced frequency resolution and reduced time resolution. The metadata frame 401D has six effective frequency bands and one effective time subframe. This provides six TF tiles.

[0095] Input signals 107 from each participant device 105 are received by the control unit 103. The input signals 107 are provided as input to the decoder 403 of the control unit 103. The decoder 403 decodes the input signals 107 and supplies the decoded input signals to the mixer 405. The mixer 405 mixes the decoded input signals to generate an audio stream which is provided as input to the encoder 407. The encoder 407 is configured to mix the audio stream to provide a downstream signal 109A which can be transmitted to the first participant device 105A.

[0096] The control unit 103 can be configured to select the mode of the audio stream 109. The selection can be based on any common mode available for each input signal 107 and / or any other preferred factors.

[0097] In the example shown in Figure 6, the control unit 103 is selected to provide different components for the audio stream. In this case, the audio stream comprises a main stream 601 and one substream 603.

[0098] In this case, the mainstream 601 includes a mix of two different audio signals. In this case, the mainstream 601 includes a mix of the upstream signal 107B from the second participant device 105B and the upstream signal 107C from the third participant device 105C. These signals have a common mode and therefore have the same TF resolution. The mainstream 601 uses the common mode. This results in a mainstream 601 having three effective frequency bands and four effective time subframes that provide 12 TF tiles.

[0099] In the example in Figure 6, substream 603 comprises an audio signal that does not share a common mode with the audio signal used for mainstream 601. In this case, substream 603 includes an upstream signal 107D from a fourth participant device 105D. Substream 603 uses the mode of the upstream signal 107D from the fourth participant device 105D. This results in substream 603 having six effective frequency bands and one effective time subframe, providing six TF tiles.

[0100] In this example, the mainstream 601 is generated from multiple input signals 107, while the substream 603 is generated from only a single input signal. The mainstream 601 may take precedence over the substream 603. The mainstream 601 can contain more data than the substream 603.

[0101] The audio stream, including the main stream 601 and the substream 603, is transmitted from the control unit 103 to the first participant device 105A. The first participant device 105A may be configured to decode the main stream 601 and the substream 603 separately from each other. In some examples, the first participant device 105A can render the decoded main stream 601 and the decoded substream 603 separately. In some examples, the first participant device 105A can combine the decoded main stream 601 and the decoded substream 603 into a combined stream and then render the combined stream.

[0102] Rendering multiple components of an audio stream can be computationally more complex than rendering an audio stream containing a single component. If participant device 105A prioritizes operating in a mode with lower complexity, the receiver can indicate this priority to the control unit 103 or any other appropriate part of the system 101. If participant device 105A indicates a priority for reducing computational complexity, the control unit 103 may select a mode that reduces this computational complexity. For example, the control unit 103 may negotiate with the other participant devices 105 to use a common mode so that the audio stream for participant device 105A can contain only one component. For example, in the example in Figure 6, if all transmitting participant devices 105B, 105C, and 105D can use a mode with a 6 × 1 TF resolution, the control unit 103 may request that this be used.

[0103] In some examples, a preferred mixing method for determining whether an audio stream contains a single component or a main stream and one or more substreams can be negotiated between the control unit 103 and the participant device 105. The negotiation may be based on the computing power and priorities of the participant device 105 and the control unit 103.

[0104] Figure 7 schematically illustrates the mixing of another audio stream. In this example, the control unit 103 receives input signals 107 from five transmitting participant devices 105. The control unit 103 may be configured to select an appropriate mode for the mixed audio stream and generate a mixed audio stream so that the mixed audio stream can be transmitted to the receiving participant devices 105A.

[0105] In Figure 7, the upstream signals 107B from the second participant device 105B, 107C from the third participant device 105C, 107D from the fourth participant device 105D, 107E from the fifth participant device 105E, and 107F from the sixth participant device 107F are received by the control unit 103. The metadata frames 401 for each of the upstream signals 107B, 107C, 107D, 107E, and 107F are shown in Figure 7. The upstream signals 107 may be in MASA format or any other suitable format.

[0106] In the example in Figure 7, the upstream signal 107B from the second participant device 105B and the upstream signal 107C from the third participant device 105C share a common mode. Figure 7 shows exemplary metadata frames 401B, 401C for the upstream signals 107B, 107C from the second participant device 105B and the third participant device 105C using this common mode. This mode has reduced frequency resolution. Metadata frames 401B, 401C have three effective frequency bands and four effective time subframes. This provides 12 TF tiles.

[0107] The upstream signal 107D from the fourth participant device 105D has a different mode from the upstream signal 107B from the second participant device 105B. The metadata frame 401D for the upstream signal 107D from the fourth participant device 105D shows reduced frequency resolution and reduced time resolution. The metadata frame 401D has six effective frequency bands and one effective time subframe. This provides six TF tiles.

[0108] The upstream signal 107E from the fifth participant device 105E and the upstream signal 107F from the sixth participant device 105F share a common mode with each other. However, this mode is different from the mode used by the other participant devices 105B, 105C, and 105D. Figure 7 shows exemplary metadata frames 401E and 401F for the upstream signals 107E and 107F from the fifth participant device 105E and the sixth participant device 105F using this common mode. This mode has reduced frequency resolution and reduced time resolution. Metadata frames 401E and 401F have three effective frequency bands and two effective time subframes. This provides six TF tiles.

[0109] Input signals 107 from each participant device 105 are received by the control unit 103. The input signals 107 are provided as input to the decoder 403 of the control unit 103. The decoder 403 decodes the input signals 107 and supplies the decoded input signals to the mixer 405. The mixer 405 mixes the decoded input signals to generate an audio stream which is provided as input to the encoder 407. The encoder 407 is configured to mix the audio stream to provide a downstream signal 109A which can be transmitted to the first participant device 105A.

[0110] The control unit 103 can be configured to select the mode of the audio stream 109. The selection can be based on any common mode available for each input signal 107 and / or any other preferred factors.

[0111] In the example in Figure 7, the control unit has been selected to provide different components for the audio stream. In this case, the audio stream comprises a main stream 601 and several substreams 603A, 603B. In this example, two substreams 603A, 603B are provided. In other examples, a different number of substreams 603 may be used.

[0112] In this case, the mainstream 601 includes a mix of two different audio signals. In this case, the mainstream 601 includes a mix of the upstream signal 107B from the second participant device 105B and the upstream signal 107C from the third participant device 105C. These signals have a common mode and therefore have the same TF resolution. The mainstream 601 uses the common mode. This results in a mainstream 601 having three effective frequency bands and four effective time subframes that provide 12 TF tiles.

[0113] In the example shown in Figure 7, the first substream 603A includes the upstream signal 107D from the fourth participant device 105D. The first substream 603A uses the mode of the upstream signal 107D from the fourth participant device 105D. This results in a substream 603 having six effective frequency bands and one effective time subframe that provide six TF tiles.

[0114] The second substream 603B includes a mix of the upstream signal 107E from the fifth participant device 105E and the upstream signal 107F from the sixth participant device 105F. These signals have a common mode and therefore have the same TF resolution. The second substream 603B uses the common mode. This results in a second substream 603B having three effective frequency bands and two effective time subframes that provide six TF tiles.

[0115] An audio stream, including a main stream 601 and several substreams 603A and 603B, is transmitted from the control unit 103 to a first participant device 105A. The first participant device 105A may be configured to decode the main stream 601 and the substreams 603A and 603B separately from each other. In some examples, the first participant device 105A can render the decoded main stream 601 and the decoded substreams 603A and 603B separately. In some examples, the first participant device 105A can combine the decoded main stream 601 and one or more of the decoded substreams 603A and 603B into a combined stream, and then render the combined stream independently of the other substreams 603.

[0116] In the examples of this disclosure, the mainstream 601 may have a higher priority than the substreams 603 in the processing order. Therefore, if the receiving participant device 105A does not have the ability to process all of the streams, the participant device 105A will process the mainstream 601 in preference to the substreams 603. In other cases, the mainstream 601 and one or more substreams 603 may be treated equally by the receiving participant device 105A. In such cases, the mainstream 601 has no priority in the processing order.

[0117] In some cases, the control unit 103 or the receiving participant device 105A, or any other suitable part of the system 101, can determine which component should be the mainstream 601 and which component should be the substream 603. In some cases, the mainstream 601 may be the component with the most mixed input signals. In some cases, the mainstream 601 may be the component with the most active input signals. In some cases, the mainstream 601 may be the component with the most reliable input signal connections.

[0118] In some examples, the control unit 103 may use other factors to select the mode of the audio stream. For example, the control unit 103 may select one or more modes based on keeping the computational complexity low. In this case, the control unit 103 may mix the input audio signals based on the TF format of each input signal.

[0119] In some examples, the control unit 103 can select one or more modes based on optimizing the output bitrate. In such cases, the control unit 103 increases the bitrate, thereby avoiding or minimizing the use of substream 603.

[0120] In some examples, the control unit 103 may select one or more modes based on negotiation with the receiving participant device 105A. The negotiation may take into account the receiving participant device 105A's priorities between bitrate, quality, and the number of decoded instances. A receiving participant device 105A indicating priority over a single decoder instance may prioritize an audio stream from the control unit 103 that contains a single component, or, if the audio stream contains multiple components, the receiving participant device 105A may decode only one of the components. For example, the receiving participant device 105A may decode only the main stream 601.

[0121] The mode selected for the input format can be negotiated during session establishment. In the case of IVAS sessions, the mode can be negotiated at the Session Description Protocol (SDP) level.

[0122] For example, a specific operating mode parameter (inf-specific-mode) may be used by each participant device 105 to indicate the modes available to the participant device 105 for a selected input format. Table 1 is an exemplary list of IVAS input formats and their corresponding modes. [Table 1] This table only shows the operating modes for the MASA and OMASA input formats. The operating modes for other input formats are not listed.

[0123] In the examples in Table 1, the MASA and OMASA input formats have two modes. These two modes include High Temporal Resolution (HTR) mode and High Frequency Resolution (HFR) mode. HTR mode uses four time subframes. HFR mode does not use subframing but can have a wider frequency bandwidth than HTR mode. These modes are examples, and different modes may be used instead of or in addition to these examples.

[0124] If a specific mode of operation parameter (inf-specific-mode) is used to indicate the available modes, a value may be assigned to that specific mode of operation parameter to indicate the available modes.

[0125] The specific operating mode parameter (inf-specific-mode) can be used to indicate the available operating modes for a selected input format. If multiple operating modes are supported within a range, they are shown separated by a hyphen (inf-specific-mode1 - inf-specific-mode2). If the operating modes are separate rather than in a contiguous range, they can be listed as comma-separated values ​​(inf-specific-mode1,inf-specific-mode2). Comma-separated values ​​can also be used if a specific operating mode is within a range but the mode priority is not the default contiguous range. In both the hyphen- and comma-separated list cases, the available operating modes may be listed in order of priority, from the highest priority mode to the lowest priority mode. The inf-specific-mode-send and inf-specific-mode-recv parameters may be used when different operating modes are used in the transmit and receive directions, respectively. The absence of inf-specific-mode may indicate that all possible input modes are available. If the selected input format can only have a single operating mode, the inf-specific-mode parameter is not required.

[0126] Another parameter (the disable-inf-specific-mode-switch parameter) can be used to restrict the transmitting participant device 105 from switching operating modes during a communication session. For this parameter, a flag can be defined to restrict mode switching. The acceptable values ​​for this parameter can be 0 and 1. If disable-inf-specific-mode-switch is 0 or does not exist, the transmitting participant device 105 is allowed to switch between the negotiated modes for the input format selected during the session. If disable-inf-specific-mode-switch is 1, the transmitting participant device 105 is not allowed to switch between the negotiated modes for the input format selected during the session.

[0127] In some examples, available modes can be indicated by using a specific value for the input format parameter (inf). Table 2 shows examples of different values ​​that can be used for the input format parameter. In this case, each input format and specific operating mode is given a unique inf parameter value. In this implementation, the information is conveyed in the inf parameter, so there is no need to use the inf-specific-mode parameter. [Table 2]

[0128] Example 1 below illustrates an exemplary SDP offer-answer negotiation for initiating a session. In this case, the sender offers a MASA input format with HTR and HFR modes. The receiver prefers the HTR mode of the MASA and includes only HTR in the SDP answer to the inf-specific-mode parameter. [Table 3]

[0129] The media line (m-line) describes the port used for the session (49152). RTP / AVP represents the RTP profile for audio and video, and 96 is an indicator of the dynamic payload type. The payload number 96 type is further described on the rtpmap-line and fmtp-line.

[0130] The rtpmap-line indicates the use of the IVAS codec with a 16kHz timestamp clock frequency. The clock frequency used for IVAS is not yet determined and may change before standardization is complete. The Enhanced Voice Services (EVS) codec can use a 16kHz clock frequency. The timestamp is one of the fields in the fixed RTP header. It is incremented throughout the session and reflects the packet flow from sender to receiver. With a 20ms speech frame block and a 16kHz timestamp clock frequency, the timestamp value increments by 320 for each consecutive frame block.

[0131] In the SDP offer, the fmtp-line parameter indicates the selected input format for the transmitting participant device 105 in the inf parameter. The numerical value refers to the inf value in Table 3 below (8 = MASA). The inf-specific-mode parameter indicates the available operating modes for the offered IVAS input format. The transmitting device 105 offers two different operating modes for MASA: HTR (High Temporal Resolution) and HFR (High Frequency Resolution). A bitrate of 512kbps is offered for the session.

[0132] The receiver sends an SDP answer to the transmitter using a modified fmtp-line. The receiver selects the preferred input format (8=MASA) for the transmitter to use. The receiver prioritizes HTR mode and includes only that in the SDP answer. Therefore, the transmitter should only use MASA's HTR input mode during the session.

[0133] In this example, the parameters br, ptime, and maxptime use the definitions presented in the Enhanced Voice Services (EVS) specification (3GPP® TS26.445). In summary, the parameters represent the following: br: Indicates the session bitrate in kilobits per second (kbps). The parameter can have either a single value (br0) or a hyphenated pair of two bitrates (br1-br2), where br1 and br2 are used as the minimum and maximum bitrates, respectively. ptime: Packet time, the length of time in milliseconds represented by the medium within the packet. In IVAS, ptime is set to 20ms. maxptime: Indicates the maximum amount of media that can be encapsulated in each packet, in milliseconds. For frame-based codecs like IVAS, the time should be an integer multiple of the frame size (20ms in the case of IVAS).

[0134] Table 3 shows the IVAS input format and its assigned attribute values. These values ​​are used in Example 1 above and Example 2 below. [Table 4]

[0135] Example 1 illustrates another exemplary SDP offer-answer scenario. In this case, the sender offers IVAS input format 8 (MASA) with two different operating modes (HTR and HFR) and a bitrate range of 13.2–160kbps. The receiver prefers the HTR mode, indicating this by listing it as the first option in the SDP answer's inf-specific-mode parameter value. In addition, the receiver wishes to avoid switching between specific operating modes during the session, indicating this with the parameter disable-inf-specific-mode-switch=1. The sender should use the HTR input format for MASA during the session. [Table 5]

[0136] Figure 8 shows another exemplary method that may be used to implement an example of the present disclosure. The method may be implemented using a system 101 as shown in Figure 1A, or using any other suitable system 101. In the exemplary method of Figure 8, some of the blocks are implemented by a control unit 103 or other similar means, and some of the blocks are implemented by participant devices 105.

[0137] In block 801, the control unit 103 receives audio signals from two or more transmitting participant devices 105. The received audio signals may contain information to be mixed into the audio stream.

[0138] In block 803, the control unit 103 receives instructions for one or more modes available for the selected input format for the transmitting participant device 105. For example, if the selected input format is MASA, the available modes may be high frequency resolution (HFR) mode and high temporal resolution (HTR) mode. Other modes may be available instead of, or in addition to, these modes.

[0139] In block 805, the control unit 103 receives a priority instruction for a mode from the receiving participant device 105. For example, the receiving participant device 105 may indicate whether it prefers HFR or HTR. In other examples, the receiving participant device 105 may indicate information that can be used by the control unit 103 to select a mode to use. For example, the receiving participant device 105 may indicate whether it prefers lower computational complexity or whether there are any other criteria to consider.

[0140] In block 807, it is determined whether there are any more input signals to be mixed. More input signals can be received from other transmitting participant devices 105. If there are more signals to be mixed, the method returns to block 801, and blocks 801-805 are repeated for the next input signal. If there are no input signals, the method proceeds to block 809.

[0141] In block 809, the control unit determines whether there is a common mode available for all input signals. If there is a common mode available, the method proceeds to block 811. If there is no common mode available for all input signals, the method proceeds to block 821.

[0142] In block 811, the method includes selecting a common mode to be used by all transmitting participant devices 105. In block 813, the control unit 103 indicates the selected mode to the participant devices 105. The control unit 103 may indicate the modes to be used to the transmitting participant devices 105. When a transmitting participant device 105 receives an instruction for the selected mode, it generates an audio signal using that mode.

[0143] In block 815, the control unit 103 mixes the received input audio signals into an audio stream. By using a common mode for all input signals, it is possible to generate a mixed audio stream without any loss of quality.

[0144] In block 817, the control unit 103 transmits the mixed audio stream to the receiving participant device 105. In block 819, the receiving participant device 105 receives the mixed audio stream and performs decoding and rendering of the mixed audio stream. This allows the audio content to be played back to the user of the receiving participant device 105.

[0145] If there is no common mode available for all input signals, the method proceeds to block 821. In this case, the audio stream is mixed into a main stream 601 and one or more substreams 603. In block 821, the control unit 103 determines whether there is a common mode for a subset of the input signals. This could be a mode common to some of the transmitting participant devices 105 but not all of them.

[0146] In block 823, the control unit 103 determines the configuration for the main stream 601 and the configuration for one or more substreams 603. The configuration for each stream may include the input signals mixed into the stream and the modes used. For example, the control unit 103 may determine that signals with a common mode may be mixed into the main stream 601, and signals without a common mode may be mixed into the substreams 603. The main stream 601 and substreams 603 can be configured so that signals using different modes are not mixed.

[0147] In block 825, the control unit 103 indicates the selected mode to the participant device 105. The control unit 103 may also indicate the mode to be used to the transmitting participant device 105. The selected mode may be the mode to be used for the mainstream 601 and one or more substreams 603. When the transmitting participant device 105 receives an instruction for the selected mode, it generates an audio signal using that mode.

[0148] In block 827, the control unit 103 mixes the received input audio signals into the mainstream 601 and one or more substreams 603. A subset of input signals sharing a common mode can be mixed into the mainstream 601, and the remaining input signals can be mixed into one or more substreams.

[0149] In block 829, the control unit 103 transmits a mixed audio stream, including a main stream 601 and one or more substreams 603, to the receiving participant device 105. In block 831, the receiving participant device 105 receives the mixed audio stream and performs decoding and rendering of the mixed audio stream. In some examples, the receiving participant device 105 can decode the main stream 601 and one or more substreams 603 separately. The receiving participant device 105 can then enable the audio content to be played back to the user of the receiving participant device 105.

[0150] Figure 9 shows an exemplary system 101 that may be used to implement an example of the present disclosure. System 101 may be an IVAS system. The system may use the MASA format or any other suitable type of format.

[0151] The exemplary system 101 in Figure 9 comprises a control unit 103 and three participant devices 105. In this example, the first participant device 105A is configured to be a receiving participant device 105A, and the second participant device 105B and the third participant device 105C are transmitting participant devices 105. It should be understood that each participant device 105 in this example of the disclosure may transmit and receive audio signals.

[0152] In this example, the control unit 103 comprises a session controller module 901, an encoder / decoder module 903, a selection module 905, and a transport stream module 907. In other examples, the control unit 103 may include different modules and / or combinations of modules.

[0153] The session controller module 901 can be configured to establish a communication session between the control unit 103 and each participant device 105. The session controller module 901 can be configured to receive session negotiation signaling from each participant device 105. The session controller module 901 can be configured to negotiate parameters for a multimedia session between the transmitting participant devices 105B, 105C and the receiving participant device 105A.

[0154] The encoder / decoder module 903 is configured to decode the audio signals received from the respective transmitting participant devices 105B and 105C. The encoder / decoder module 903 is also configured to encode the mixed audio stream for transmission to the receiving participant device 105A. The encoder / decoder module 903 may be equipped with a MASA encoder / decoder or any other suitable type of encoder.

[0155] The selection module 905 may be configured to select the mode to be used for the mixed audio stream. The selection module 905 can receive information indicating the modes available for the selected input format for each transmitting participant device 105. The selection module 905 can also receive information regarding the priority of the receiving participant device 105A, such as computational requirements. The selection module 905 can select the mode to be used for mixing using any of the methods or processes described herein.

[0156] The transport stream module 907 can be configured to generate a transport audio stream. The transport stream can use any suitable protocol, such as the Real-Time Transport Protocol (RTP). The transport stream module 907 provides an encoded stream as a payload.

[0157] Each participant device 105 also comprises a microphone array 909, a session controller module 911, an encoder / decoder module 913, a processing module 915, and a transport stream module 917. In other examples, the control unit 103 may include different modules and / or combinations of modules.

[0158] The microphone array 909 may comprise multiple microphones. The microphones are configured to capture sound and generate electrical output signals. The microphones within the microphone array 909 may be spatially arranged to allow spatial audio to be captured.

[0159] The participant device 105 is configured so that the output signal from the microphone array 909 is provided to the processing module 915. The processing module 915 is configured to process the microphone signal into a selected format. The processing module 915 is configured to process the microphone signal into a selected mode of the selected format.

[0160] The participant device 105 is configured such that the output signal from the processing module 915 is provided as input to the encoder module 913. The encoder module 913 is configured to encode the processed microphone signal and send it to the control unit 103. The encoder module 913 may be a MASA encoder or any other suitable type of encoder.

[0161] The encoded signal is provided to the transport stream module 917. The transport stream module 917 may be configured to generate a transport audio stream. The transport stream can use any suitable protocol, such as the Real-Time Transport Protocol (RTP). The transport stream module 917 provides the encoded stream as a payload. The encoded stream is transmitted to the control unit 103.

[0162] The session controller module 911 can be configured to establish a communication session with the control unit 103. The session controller module 911 can be configured to transmit session negotiation signaling to the control unit 103. The session controller module 911 can be configured to negotiate parameters for a multimedia session between the participant device 105 and the control unit 103. The session controller module 911 of the participant device 105 is configured to send signals to the session controller module 911 of the control unit 103.

[0163] Participant devices may also be configured to receive head tracking information. Head tracking information can be received from one or more positioning devices or sensors configured to monitor user 111. This information may be used by participant device 105 to determine the orientation of the user's head and to render the received audio stream.

[0164] In the example in Figure 9, the participant device 105 is identical. In other examples, the participant device 105 may differ.

[0165] Figure 10 schematically shows a device 1001 that may be used to implement an example of the present disclosure. In this example, the device 1001 comprises a controller 1003. The controller 1003 may be a chip or a chipset. The device 1001 may be located within a control unit 103 or participant device 105 or any other suitable device.

[0166] In the example shown in Figure 10, the controller 1003 can be implemented as a controller circuit. In some examples, the controller 1003 may be implemented as hardware only, or it may have a specific form of software including only firmware, or it may be a combination of hardware and software (including firmware).

[0167] As shown in Figure 10, the controller 1003 can be implemented by using executable instructions of a computer program 1009 in a general-purpose or dedicated processor 1005, which can be stored on a computer-readable storage medium (disk, memory, etc.) executed by such a processor 1005, using instructions that enable hardware functionality.

[0168] The processor 1005 is configured to read from and write to memory 1007. The processor 1005 may also have an output interface to which data and / or commands are output by the processor 1005, and an input interface to which data and / or commands are input to the processor 1005.

[0169] Memory 1007 stores computer program 1009, which includes computer program instructions (computer program code 1011) that control the operation of controller 1003 when loaded into processor 1005. The computer program instructions of computer program 1009 provide logic and routines that enable controller 1003 to perform the actions shown in the attached diagram. Processor 1005 can load and execute computer program 1009 by reading memory 1007.

[0170] The device 1001 includes at least one processor 1005 and at least one memory 1007 for storing instructions, and when the instruction is executed by at least one processor 1005, it causes the device 1001 to perform at least the steps of: obtaining instructions for one or more modes available for a selected input format of a first audio signal (501), wherein the first audio signal is received from a first source; obtaining instructions for one or more modes available for a selected input format of a second audio signal (503), wherein the second audio signal is received from a second source, and the first audio signal and the second audio signal are combined to form an audio stream; selecting a mode for the audio stream including the first audio signal and the second audio signal (505), wherein the selecting step is at least partially based on one or more common modes available for the selected input format of the first audio signal and the selected input format of the second audio signal; and transmitting instructions for the selected mode to the respective sources (507).

[0171] As shown in Figure 10, the computer program 1009 can reach the controller 1003 via any suitable delivery mechanism 1013. The delivery mechanism 1013 can be, for example, a machine-readable medium, a computer-readable medium, a non-temporary computer-readable storage medium, a computer program product, a memory device, a recording medium such as a compact disc read-only memory (CD-ROM) or a digital multipurpose disc (DVD), or solid memory, or a manufactured product containing or tangibly embodying the computer program 1009. The delivery mechanism can be a signal configured to reliably transfer the computer program 1009. The controller 1003 can propagate or transmit the computer program 1009 as a computer data signal. In some examples, the computer program 1009 can transmit to the controller 1003 using a wireless protocol such as Bluetooth®, Bluetooth® Low Energy, Bluetooth® Smart, 6LoWPan (IPv6 over Low power personal area networks), ZigBee®, ANT+, Near Field Communication (NFC), Radio Frequency Identification (RFID), Wireless Local Area Network (Wi-Fi), or any other suitable protocol.

[0172] The computer program 1009 includes a computer program instruction causing the device 1001 to perform at least the steps of: obtaining instructions for one or more modes available for a selected input format of a first audio signal (501), wherein the first audio signal is received from a first source; obtaining instructions for one or more modes available for a selected input format of a second audio signal (503), wherein the second audio signal is received from a second source, and the first audio signal and the second audio signal are combined to form an audio stream; selecting a mode for an audio stream including the first audio signal and the second audio signal (505), wherein the selecting step is at least partially based on one or more common modes available for a selected input format of the first audio signal and a selected input format of the second audio signal; and transmitting instructions for the selected mode to the respective sources (507).

[0173] Computer program instructions may be included in computer program 1009, non-temporary computer-readable media, computer program products, and machine-readable media. In some, but not all, computer program instructions may be distributed across multiple computer programs 1009.

[0174] Although memory 1007 is illustrated as a single component / circuit, it can be implemented as one or more separate components / circuits, some or all of which can be internal / removable and / or provide permanent / semi-permanent / dynamic / cache storage.

[0175] Although processor 1005 is shown as a single component / circuit, it can be implemented as one or more separate components / circuits, some or all of which can be integrated / removable. Processor 1005 may be a single-core or multi-core processor.

[0176] Terms such as "computer-readable storage medium," "computer program product," "tangibly embodied computer program," or "controller," "computer," and "processor" should be understood to encompass not only computers with different architectures such as single / multiprocessor architectures and sequential (Von Neumann) / parallel architectures, but also specialized circuits such as field-programmable gate arrays (FPGAs), application-specific circuits (ASICs), signal processing devices, and other processing circuits. References to computer programs, instructions, code, etc., should be understood to encompass programmable content for hardware devices, whether it be software or firmware for programmable processors, such as instructions for a processor, or configuration settings for fixed-function devices, gate arrays, or programmable logic devices.

[0177] As used in this application, the term "circuit" may refer to one, more, or all of the following: (a) Hardware-only circuit implementation (e.g., implementation using only analog and / or digital circuits), (b) A combination of hardware circuitry and software, for example (where applicable), (i) combinations of analog and / or digital hardware circuits and software / firmware, (ii) A part of a hardware processor in which software (including a digital signal processor), software, and one or more memories work together to enable a device such as a mobile phone or server to perform various functions. (c) Hardware circuits and / or processors, such as microprocessors, that require software (e.g., firmware) for operation, but may not be present if the software is not required for operation.

[0178] This definition of circuit applies to all use of the term in this application, including any claim. As a further example, as used in this application, the term circuit also encompasses implementations of hardware circuits or processors and their associated software and / or firmware. The term circuit also encompasses, for example, a baseband integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device, as applicable to the elements of a particular claim.

[0179] The blocks shown in Figures 4 and 8 can represent steps and / or sections of code in a method in computer program 1009. The examples of specific orders for the blocks do not necessarily suggest that there is a required or preferred order for the blocks, and the order and arrangement of the blocks can be changed. Furthermore, some blocks may be omitted.

[0180] The term “comprise” is used herein in a comprehensive, not exclusive, sense. That is, any reference to X containing Y indicates that X may contain only one Y or multiple Ys. If “comprise” is intended to be used in an exclusive sense, it will be evident in the context by referring to “comprising only one” or by using “consisting.”

[0181] In this explanation, the terms “connection,” “combination,” and “communication,” as well as their derivatives, mean to be operationally connected / combined / communicated. It should be understood that any number or combination of intervening components (including no intervening components) may exist, i.e., providing direct or indirect connection / combination / communication. Any such intervening components may include hardware and / or software components.

[0182] As used herein, the terms “determine” (and its grammatical variations) include, in particular, calculation, operation, processing, derivation, measurement, investigation, identification, retrieval (e.g., retrieving a table, database or another data structure), confirmation, etc. “Determine” may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), retrieval, etc. Furthermore, “determine” may also include resolving, selecting, choosing, establishing, etc.

[0183] This explanation refers to various examples. Descriptions of features or functions relating to an example indicate that those features or functions are present in that example. The use of the terms “example,” “for example,” “can,” or “may” in the text, whether explicitly stated or not, indicates that such features or functions are present in at least the example being described, whether or not they are described as examples, and that they may, but not necessarily, be present in some or all other examples. Thus, “example,” “for example,” “can,” or “may” refer to a specific instance in the example class. Properties of an instance may be properties of that instance only, or properties of the class, or properties of a subclass of the class that includes some, but not all, instances within the class. Thus, it is implicitly disclosed that features described in one example but not in another may, where possible, be used as part of a working combination in other examples, but are not necessarily required to be used in other examples.

[0184] Examples have been described in the preceding paragraphs with reference to various embodiments, but it should be understood that modifications to the given embodiments may be made without departing from the scope of the claims.

[0185] The features described above may be used in combinations other than those explicitly described above.

[0186] While the functions are described by referring to specific features, these functions may be performable by other features, whether or not they are described.

[0187] While we have described the features by referring to specific examples, these features may also exist in other examples, whether or not they are described.

[0188] The terms “a,” “an,” or “the” are used herein in an inclusive sense, not an exclusive sense. That is, any reference to X containing a / an / the Y indicates that X may contain only one Y or more Ys, unless the context clearly indicates otherwise. If “a,” “an,” or “the” is intended to be used in an exclusive sense, it will be evident in the context. In some situations, the use of “at least one” or “one or more” may be used to emphasize an inclusive meaning, but the absence of these terms should not be interpreted as inferring any exclusive meaning.

[0189] The presence of a feature (or combination of features) in a claim is a reference to the feature or (combination of features) itself, and to features (equivalent features) that achieve substantially the same technical effect. Equivalent features include, for example, variations of features that achieve substantially the same result in substantially the same way. Equivalent features include, for example, features that perform substantially the same function in substantially the same way to achieve substantially the same result.

[0190] This explanation refers to various examples in which adjectives or adjective phrases are used to describe the characteristics of the examples. Such descriptions of characteristics of the examples indicate that the characteristics are present in some examples exactly as described and in others substantially as described.

[0191] While the above description illustrates some examples of the present disclosure, those skilled in the art will be aware of possible alternative structures and methods that provide equivalent functionality to certain examples of such structures and methods described above and have been omitted from the above description for the sake of brevity and clarity. Nevertheless, the above description should be read as implicitly including references to such alternative structures and methods that provide equivalent functionality, unless such alternative structures or methods are expressly excluded in the above description of the examples of the present disclosure.

[0192] While the foregoing specification attempts to draw attention to these features considered important, it should be understood that the applicant may seek protection through the claims with respect to any patentable feature or combination of features mentioned herein and / or shown in the drawings, whether or not they are emphasized.

Claims

1. A device comprising at least one processor and at least one memory for storing instructions, wherein when an instruction is executed by the at least one processor, the device contains at least one A step of obtaining an indication of one or more available modes for a selected input format of a first audio signal, wherein the first audio signal is received from a first source; A step of obtaining an instruction for one or more available modes for a selected input format of a second audio signal, wherein the second audio signal is received from a second source, and the first audio signal and the second audio signal are combined to form an audio stream. A step of selecting a mode for the audio stream comprising the first audio signal and the second audio signal, wherein the selecting step is at least in part based on one or more common modes available for the selected input format of the first audio signal and the selected input format of the second audio signal. An apparatus characterized by performing the step of sending instructions for the selected mode to each source.

2. The apparatus according to claim 1, wherein the indication of the available modes is received for two or more audio signals from different sources, the different audio signals may have different modes available for a selected input format, and the audio signals are combined to form an audio stream.

3. The apparatus according to claim 1, characterized in that the selected mode includes a mode common to two or more audio signals from two or more sources.

4. The aforementioned mode is at least, The mode most commonly available for the selected input format with respect to the aforementioned source, For the selected input format, the computational load for mixing the audio signals from the source into a combined stream in each available mode, For the selected input format, in each of the available modes, a computational load for decoding the audio signal from the source and / or a computational load for encoding the mixed audio stream, The apparatus according to claim 1, characterized in that it is selected based in part on at least one of the following.

5. The apparatus according to claim 1, characterized in that, in the selected mode, the apparatus performs the steps of mixing at least the first audio signal and at least the second audio signal using a common mode to generate a mixed audio stream, and enabling the transmission of the mixed audio stream.

6. The apparatus according to claim 1, characterized in that the selected mode includes a first mode for the audio mainstream and a second mode for the audio substream.

7. The apparatus according to claim 6, characterized in that the mainstream includes mixing two or more audio signals having a common mode in the selected input format.

8. The apparatus according to claim 6, characterized in that the apparatus performs the step of enabling transmission of the main stream and the substream without mixing the main stream and the substream.

9. The apparatus according to claim 1, characterized in that it performs the steps of receiving an instruction for a preferred mode for an end-user device, and using the instruction for the preferred mode to assist in selecting the mode for the audio stream.

10. The apparatus according to claim 1, characterized in that the aforementioned audio signal includes a metadata-assisted spatial audio signal.

11. The apparatus according to claim 1, characterized in that the selected input format includes a metadata-assisted spatial audio format.

12. The apparatus according to claim 1, characterized in that one or more modes available for an input format have time-frequency resolution.

13. The apparatus according to claim 12, characterized in that at least one of the available modes has higher frequency resolution, and at least one of the available modes has higher temporal resolution.

14. The apparatus according to claim 1, characterized in that it performs the step of generating spatial metadata using the selected mode.

15. A step of obtaining an indication of one or more available modes for a selected input format of a first audio signal, wherein the first audio signal is received from a first source; A step of obtaining an instruction for one or more available modes for a selected input format of a second audio signal, wherein the second audio signal is received from a second source, and the first audio signal and the second audio signal are combined to form an audio stream. A step of selecting a mode for the audio stream comprising the first audio signal and the second audio signal, wherein the selecting step is at least in part based on one or more common modes available for the selected input format of the first audio signal and the selected input format of the second audio signal. A method characterized by including the step of sending instructions for the selected mode to each source.

16. The method according to 15, wherein the indication of the available modes is received for two or more audio signals from different sources, the different audio signals may have different modes available for a selected input format, and the audio signals are combined to form an audio stream.

17. The aforementioned mode is at least, The mode most commonly available for the selected input format with respect to the aforementioned source, For the selected input format, the computational load for mixing the audio signals from the source into a combined stream in each available mode, The method according to 15, characterized in that it is selected in part on the basis of at least one of the computational loads of decoding the audio signal from the source and / or encoding the mixed audio stream in each of the available modes for the selected input format.

18. The method according to 15, further comprising the steps of mixing at least the first audio signal and at least the second audio signal using a common mode to generate a mixed audio stream in the selected mode, and enabling transmission of the mixed audio stream.

19. The method according to 15, characterized in that one or more modes available for an input format have time-frequency resolution.

20. The method according to 19, characterized in that at least one of the available modes has higher frequency resolution, and at least one of the available modes has higher temporal resolution.

Images