The method of creating a data transmission format for audio, communication terminals, and storage media that can be read by a processor.

Abstract

The invention relates to a method for generating an audio data transmission format in a real-time communication session between communication devices (A, B), wherein the data transmission format for the Extended Audio Encoder-Decoder (IVAS), extending the single-channel audio encoder-decoder (EVS), reusing the data transmission format of the aforementioned single-channel audio encoder-decoder, and wherein at least one existing unused or reserved field of the data transmission format of the aforementioned single-channel audio encoder-decoder is used to signal the Extended Audio Encoder-Decoder and / or transmit at least information about the bit rate of the Extended Audio Encoder-Decoder. The invention also relates to a communication terminal implementing the method for generating the aforementioned data transmission format and a storage medium readable by a processor.

Description

Optimized format for transporting audio data from a voice over IP communication session

[0001] The present invention relates to the field of telecommunications and more particularly to packet communication networks. In this type of network, it is possible to route data streams associated with real-time services.

[0002] The Internet Protocol, called IP for "Internet Protocol", developed by the IETF, for "Internet Engineering Task Force", is implemented on packet communication networks to support both non-real-time services such as data transfer services, web page consultation, electronic messaging, as well as real-time or conversational services, such as IP telephony, IP video telephony or IP video broadcasting.

[0003] The invention relates more particularly to the generation and reception of payload, also called transport format, for the transport of data coded according to the RTP protocol (for "Real-Time Transport Protocol" in English) during real-time communication between two communication devices, for example between two communication terminals or between a communication terminal and network equipment, to process real-time signals such as voice or video signals.

[0004] Reference is made to IETF specifications RFC 3550 and RFC 4566 for the known state-of-the-art bases on the RTP protocol as well as the SDP (Session Description Protocol) protocol.

[0005] The payload format for transporting data using the RTP protocol, also called transport format in the rest of the document, is defined for the EVS codec in the 3GPP TS 26.445 Annex A specification.

[0006] It should be remembered that the EVS codec (for “Enhanced Voice Services” in English) was standardized by the 3GPP organization at the end of 2014. It is currently defined in three source code versions: • Fixed point source code (3GPP TS 26.442)

[0007] • Floating point source code (3GPP TS 26.443)

[0008] • Alternative floating point source code version (3GPP TS 26.452)

[0009] It works with 20 ms frames and at different sampling rates (8, 16, 32, 48 kHz) for mono signals. Stereo or multichannel signals can be encoded in multi-mono only, i.e. by encoding the channels separately, without possible metadata. The EVS codec defines 4 types of audio bands:

[0010] - narrow band called Narrowband (NB)

[0011] - wideband (WB)

[0012] - super-wideband (SWB)

[0013] - full band called Fullband (FB)

[0014] It includes two operating modes: "Primary" mode: in this case it operates at a constant bit rate of 7.2 kbit / s to 128 kbit / s and also includes a special mode called VBR with an average bit rate of 5.9 kbit / s - these bit rates are designated by "EVS primary modes" (or "primary modes"). Backward compatible mode with AMR-WB called "AMR-WB IO": in this case it operates at AMR-WB bit rates, i.e. 6.6, 8.85, 12.65, 14.25, 15.85, 18.25, 19.85, 23.05, 23.85 kbit / s.

[0015] Additionally, the EVS codec allows for optimized coding of silences by discontinuous transmission (DTX), which includes voice activity detection (VAD) and comfort noise generation (CNG). When DTX is enabled, inactive frames are encoded by Silence Insertion Descriptor (SID) frames, which are typically sent every 160 ms.

[0016] Additionally, EVS includes a special channel-aware mode (CAM) limited to 13.2 kbit / s in WB and SWB, which uses partial redundancy to improve robustness to frame losses.

[0017] The transport format for transporting EVS-encoded data over RTP is defined in Annex A of TS 26.445. This format is illustrated in Figures 1a to 1i.

[0018] There are two different modes for forming EVS packets:

[0019] • The default mode: packets are either in "Compact" format or in "Header-full" format. To avoid size conflicts between these two modes, priority is given to Compact mode, whose sizes are "protected", while the sizes of "Header-full" mode are penalized by adding zero padding in the event of a conflict with a size of "Compact" mode.

[0020] • The "Header-full" mode which consists of adding payload headers (one CMR byte for "Codec Mode Request" according to the SDP "cmr" parameter, one or more ToC bytes for "Table of Content" to describe the multiplexed frames and be able to extract them) before the coded data which correspond to one or more coded frames.

[0021] Examples of RTP payload in "Compact" format are given in Figures 1a and 1b for the EVS Primary and EVS AMR-WB IO cases, respectively; we observe in particular that for EVS Primary the payload is limited to the SD (Speech data) encoded data, without any transport header and without any possibility to signal an adaptation request (); for EVS AMR-WB IO, the data encoded in "Compact" mode are reorganized (some bits are swapped) and a CMR (Codec Mode Request) field reduced to 3 bits is defined as header (). This particular case of 3-bit CMR is not detailed here.

[0022] Examples of RTP payloads in header-full format are given in Figures 1c and 1d. In the first case (), a single frame is transported and a ToC (Table of Content) header byte (8 bits) is inserted. In the second case (), two frames are transported in the same packet with 2 ToC headers (2 bytes) and a CMR field (1 byte) is also used.

[0023] A detailed definition of these fields is given in specification TS 26.445, Annex A. For the sake of brevity, all the details of this specification are not repeated here.

[0024] The choice of payload mode (either default or header-full) is controlled by the EVS codec's SDP (Session Description Protocol) parameter called "hf-only".

[0025] We do not review here the complete list of SDP parameters of the EVS codec (see 3GPP TS 26.445 Annex A), but we remind you that there are global parameters, parameters specific to the EVS Primary mode (e.g.: "br" or "bw") and parameters specific to EVS AMR-WB IO (e.g.: "mode-set").

[0026] It is possible to adapt the EVS codec by Codec Mode Request (CMR), which is a form of in-band signaling, i.e. in the RTP stream. CMR is not possible in "compact" format except for EVS AMR-WB IO (with restrictions).

[0027] For EVS, in header-full mode, the CMR byte (8 bits) is divided into three parts represented in:

[0028] • H (1 bit): always 1 (specifying a CMR request)

[0029] • T (3 bits): bits indicating the type of request to distinguish the different EVS Primary modes in NB, WB, SWB, FB (including VBR, CAM modes) and EVS AMR-WB IO.

[0030] • D (4 bits): CMR request

[0031] The possible values ​​of the CMR field are defined in the TS 26.445 specification, Annex A and specify the different modes that the codec can take. They are recalled in. We can observe that some values ​​are indicated as not used 'Not Used' or reserved 'Reserved' and a value indicated at 'NO_REQ' is an empty CMR request.

[0032] Similarly for EVS the ToC byte (8 bits) is divided into 3 parts represented in:

[0033] • H (1 bit): always 0 (to distinguish the byte as a ToC field and not a CMR)

[0034] • F (1 bit): If F=1, another frame follows the current frame, if F=0, it is the last frame of the packet.

[0035] • FT (6 bits): bits indicating the “Frame type” either EVS Primary or EVS AMR-WB IO (including SID mode).

[0036] This field is itself divided into 3 parts defined in the tables shown in figures 1h and 1i and as defined in tables A.4 and A.5 of the 3GPP TS 26.445 specification: EVS mode (1 bit), Unused / Q bit (1 bit) depending on the EVS mode bit, and EVS rate (4 bits).

[0037] Note that for EVS it is also possible to add bit stuffing (padding bytes) to the end of the payload. In this case the "P" bit (for "Padding") in the RTP header can be - optionally - set to 1 and the additional bits must be added; the last byte of this "padding" contains the number of bytes added to the end of the payload (including this last byte).

[0038] The IVAS (Immersive Voice and Audio Services) codec is currently being defined. A high-level description of the current state of the codec is given in 3GPP TS 26.250 V1.0.0 dated 2023-09-15.

[0039] The IVAS codec is an extension of the EVS codec to represent stereo and immersive formats. It is identical to EVS for mono encoding.

[0040] For stereo and immersive formats, the IVAS codec bitrate ranges from 13.2 to 512 kbit / s (with restrictions depending on the exact format).

[0041] The different input formats of the IVAS encoder are defined in Table 1 below:

[0042] Format TypeInput Format NameDescriptionmonoMONOMono (EVS)2 channelsSTEREOStereoBINAURALBinauralISM (Independent Stream with Metadata)ISM11 mono object with metadataISM22 independent mono objects with metadataISM33 independent mono objects with metadataISM44 independent mono objects with metadataSBA (scene-based audio)FOAAmbisonic order 1HOA2Ambisonic order 2HOA3Ambisonic order 3MC (multichannel)MC515.1 multichannelMC717.1 multichannelMC5125.1+2 multichannelMC5145.1+4 multichannelMC7147.1+4 multichannelMASA (Metadata Assisted Spatial Audio)MASA11-channel MASA (metadata-assisted spatial audio) formatMASA21-channel MASA (metadata-assisted spatial audio) format 2 channelsOSBA (object-based audio with scene-based audio)OSBACombination of FOA / HOA2 / HOA3 and ISMOMASA (object-based audio with metadata-assisted spatial audio)OMASACombination of MASA and ISM

[0043] It is not excluded that other coding modes will be introduced later in the development of the codec, for example for immersive rendering applications divided between pre-rendering in the network (edge ​​or cloud) and broadcasting of a pre-rendered signal to a terminal using at least one other format of the IVAS codec which is not listed in Table 1.

[0044] The bitrate of the IVAS codec actually depends on the input format (in bit / s):Mono: same as EVS Stereo: 13200, 16400, 24400, 32000, 48000, 64000, 80000, 96000, 128000, 160000, 192000, 256000 ISM: ISM1: 13200, 16400, 24400, 32000, 48000, 64000, 80000, 96000, 128000 ISM2: 16400, 24400, 32000, 48000, 64000, 80000, 96000, 128000, 160000, 192000, 256000ISM3: 24400, 32000, 48000, 64000, 80000, 96000, 128000, 160000, 192000, 256000, 384000ISM4: 24400, 32000, 48000, 64000, 80000, 96000, 128000, 160000, 192000, 256000, 384000SBA, MASA, MC: 13200, 16400, 24400, 32000, 48000, 64000, 80000, 96000, 128000, 160000, 192000, 256000, 384000, 512000

[0045] If additional formats are added to Table 1, it can be assumed that the bitrate would be defined among these rates ranging from 13200 to 512000 bit / s.

[0046] The different output formats of the IVAS decoder / renderer are defined in Table 2 below:

[0047] Format TypeInput Format NameDescriptionmonoMONOMonostereoSTEREOStereobinauralBINAURALBinaural (with / without reverb)MC (Multichannel)5_15.1 Multichannel7_17.1 Multichannel5_1_25.1+2 Multichannel5_1_45.1+4 Multichannel7_1_47.1+4 MultichannelcustomCustom speaker configurationSBA (Scene-Based Audio)FOAAmbisonic 1st orderHOA2Ambisonic 2nd orderHOA3Ambisonic 3rd orderExternalExternalProprietary format for an external renderer

[0048] It is important to note that the algorithmic delay of the IVAS codec depends on the encoding / decoding format, it is 32 ms for the stereo format and it ranges from 32 to 38 ms for other input formats than stereo.

[0049] The payload format for transporting data via the RTP protocol, also called the transport format ("payload format") for the IVAS codec, is not yet defined.

[0050] Two proposals have recently been published: A first proposal is given in the document at the following link, dated “September 7, 2023:

[0051] https: / forge.3gpp.org / rep / ivas-codec-pc / ivas-codec / - / wikis / uploads / 5d3146bd2f9548a609c3f2d1ef431363 / ivas-rtp-payload-format.pdf

[0052] This first proposal assumes that the transport format of the IVAS codec is different from EVS. In this proposal, no compact mode is proposed in order to keep only a "header-full" mode and thus avoid complications and dependencies on an SDP parameter.

[0053] To be able to switch between mono mode (EVS) and immersive mode (IVAS) in the same call, according to this proposal, it would be necessary to negotiate both transport formats at the same time, which is not possible in multimedia services over IMS (for "IP Multimedia SubSystem" in English).

[0054] Furthermore, in an asymmetric communication where the uplink sends for example a mono coded signal and the downlink receives an immersive coded signal, as in a spatial conference bridge scenario, the first proposition implies that two different codecs are instantiated in each direction (EVS and IVAS) from a transport format point of view ("payload format").

[0055] A second proposal is given in the document at the following link, dated “September 11, 2023:

[0056] https: / forge.3gpp.org / rep / ivas-codec-pc / ivas-codec / - / wikis / uploads / 23f8cb74ba3631300f44804d2640f46f / Nokia-IVAS-RTP-Payload-draft-v1.pdf

[0057] This second proposal defines a compact mode for IVAS, which implies a list of protected packet sizes. However, if the compact mode were also retained for the mono format of IVAS, which corresponds to EVS, there would be a conflict between the EVS Primary (mono) and IVAS packet sizes; additional zero-padding rules would have to be added to either EVS Primary or IVAS, which is not attractive.

[0058] This proposal does not define any CMR for IVAS, and relies on the RTCP protocol (for "Real-Time Control Protocol") to convey adaptation requests, while voice over IMS relies on the RTP profile called AVP (for "Audio Video Profile" in English) and not AVPF (for "Audio Video Profile with Feedback" in English). Thus the use of RTCP (RTCP-APP or RTCP-FB) is not possible in this context of multimedia services over IMS.

[0059] This second proposal changes the ToC (Table of Content) byte format defined for EVS (TS 26.445 Annex A) to include in the case of IVAS additional signaling information and a 4-bit flow indication.

[0060] It should be noted that even though mono format seems to be considered, the bitrates associated with EVS are not explicitly listed, which implies that mono (EVS) format is not really transported in compact mode. This proposal also adds a new idea of ​​transporting so-called "processing information" frames to convey non-audio information such as the orientation of a terminal, the importance of a format.

[0061] Thus, existing transport format proposals for the IVAS codec are not compatible with voice or multimedia service transport applications over IMS, which can be problematic for deploying codecs in communication services.

[0062] Furthermore, these different proposals do not allow easy switching in the same call between a mono mode using an EVS-type codec and an immersive mode using a single IVAS-type codec, without having to disconnect and restart a call. They do not ensure continuity of service between the mono format (EVS) and the stereo / immersive formats (IVAS).

[0063] The invention improves the state of the art.

[0064] To this end, the invention relates to a method for generating an audio data transport format during a communication session between communication devices, in which the data transport format for an extended audio codec, extending a mono audio codec, reuses the data transport format of said mono audio codec, and where at least one existing, unused or reserved field of the data transport format of said mono audio codec is used to signal the extended audio codec and / or to transmit at least information of the bit rate of the extended audio codec.

[0065] In the same way, the invention relates to a method for receiving audio data transport format during a real-time communication session between communication devices, in which the data transport format for an extended audio codec extending a mono audio codec reuses the data transport format of said mono audio codec, and where at least one existing, unused or reserved field of the data transport format of said mono audio codec is used to signal the extended audio codec and / or to transmit at least information on the bit rate of the extended audio codec.

[0066] Thus, a single data format is specified for both the IVAS extended codec and the EVS codec, which allows on the one hand to easily switch from one codec to another during the same call, in a way compatible with multimedia services on IMS which can only select a single transport format and negotiate only a single transport format.

[0067] According to a first embodiment, an unused and existing field of the data transport format of said mono audio codec is used to signal the extended audio codec and another field is used to transmit at least information of the bit rate of the extended audio codec.

[0068] This allows for maximum compatibility with the existing mono format when switching by keeping the meaning of all other preset values ​​intact, and thus maximizing backward compatibility with the existing mono format.

[0069] In a second embodiment, at least one existing, unused or reserved codec adaptation request field of the mono audio codec is used to convey at least information of the bit rate of an extended codec to be used.

[0070] In a particular embodiment, each of the possible bitrates of the extended codec is indicated in an existing, unused or reserved, codec adaptation request field of the mono codec.

[0071] Again, this allows for maximum compatibility with the existing mono format when requesting to switch from stereo / immersive to mono, keeping the meaning of all other predefined CMR values ​​intact, and thus maximizing backward compatibility with the existing mono format. This implementation uses only a single byte for the CMR request.

[0072] In an alternative embodiment, a second adaptation request specifying at least one format and / or one audio band of the extended codec is inserted following said request.

[0073] In a third embodiment, an existing unused or reserved codec adaptation request field of the mono audio codec is used to inform of the existence of a second code specifying a format and / or bitrate of the extended codec to be used.

[0074] Similarly, an existing unused or reserved codec adaptation request field of the mono audio codec is used to inform of the existence of a second code specifying a format and / or bitrate of the extended codec used in transmission.

[0075] The advantages are similar to the second embodiment, with however a disadvantage which makes this third mode less advantageous in terms of additional flow, on the other hand makes it more general by the use of an additional byte which makes it possible to signal a change of immersive format.

[0076] The invention relates to a communication terminal comprising a processing circuit capable of implementing a method for generating an audio data transport format during a real-time communication session with a communication device, in which the data transport format for an extended audio codec, extending a mono audio codec, reuses the data transport format of said mono audio codec, at least one existing, unused or reserved field of the data transport format of said mono audio codec being used to signal the extended audio codec and / or transmit at least information on the bit rate of the extended audio codec.

[0077] The invention also relates to an audio data transport format for an extended audio codec reusing a data transport format of a mono audio codec in which at least one existing, unused or reserved field of the data transport format of said mono audio codec is used to signal the extended audio codec and / or to transmit at least information of the bit rate of the extended audio codec.

[0078] Such a transport format is in accordance with any of the embodiments described above.

[0079] The invention also relates to a computer program comprising instructions for implementing the generation method according to the invention, according to any one of the particular embodiments described above, when said program is executed by a processor.

[0080] Such instructions can be stored permanently in a non-transitory memory medium of the communication terminal implementing the transport method according to the invention.

[0081] This program may use any programming language, and may be in the form of source code, object code, or code intermediate between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0082] The invention also relates to a recording medium or information medium readable by a computer, and comprising instructions of a computer program as mentioned above.

[0083] The recording medium may be any entity or device capable of storing the program. For example, the medium may include a storage medium, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording medium, for example a mobile medium, a hard disk drive or an SSD.

[0084] On the other hand, the recording medium may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means, so that the computer program contained therein is remotely executable. The program according to the invention may in particular be downloaded over a network, for example an Internet-type network.

[0085] Alternatively, the recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to perform or to be used in performing the aforementioned transport method.

[0086] According to an exemplary embodiment, the present technique is implemented by means of software and / or hardware components.

[0087] Other characteristics and advantages of the invention will appear more clearly on reading the following description of particular embodiments, given as simple illustrative and non-limiting examples, and the appended drawings, among which:

[0088] ,,,,,,,and illustrate the transport format defined for the EVS codec, described previously;

[0089] illustrates an embodiment of a voice over IP communication system and a method of generating a data transport format according to an embodiment of the invention as well as a method of receiving such a data transport format;

[0090] ,,and illustrate a first embodiment according to the invention of a transport format adapted to the IVAS extended codec extending an EVS mono audio codec, and to the EVS mono codec by using an unused and existing field of the data transport format of the mono audio codec to signal the extended audio codec and by using another field to transmit at least information of the bit rate of the extended audio codec;

[0091] ,,,and illustrate a second and third embodiment according to the invention of a transport format adapted to the IVAS extended codec and to the EVS mono codec by the use of at least one existing, unused or reserved, codec adaptation request field of the mono audio codec to transmit information of the bit rate of the extended codec to be used;

[0092] illustrates a fourth embodiment according to the invention of a transport format adapted to the IVAS extended codec and to the EVS mono codec by the use of at least one existing codec adaptation request field, unused or reserved, of the mono audio codec to inform of the existence of a second code specifying a format and / or a bit rate of the extended codec to be used in reception or transmission;

[0093] and illustrate the extended coding format codes and the flow codes specified by this second code depending on the reception or transmission;

[0094] illustrates a fourth embodiment according to the invention of a transport format adapted to the IVAS extended codec and to the EVS mono codec by the use of the fields as defined in la and in la ; and

[0095] illustrates a hardware embodiment of a terminal implementing the method for generating a data transport format according to one embodiment as well as a method for receiving such a data transport format.

[0096] The invention will now be described below with reference to Figures 2 to 7.

[0097] Illustrates an example of a two-way communication system between two communication devices A and B, for example two communication terminals A and B or a communication terminal A and a network device B (or vice versa).

[0098] The communication devices implement a method for generating an audio data transport format according to one embodiment of the invention. The ambient acoustic signal is for example picked up by one or more microphones (101 and 151) on each side of the communication.

[0099] The remote signal is reproduced on one or more loudspeakers (102 and 152). The audio signals captured and reproduced generally undergo different acoustic treatments in transmission (101 and 151) and reception (103 and 153) such as for example:

[0100] • analog / digital conversion, gain control, noise reduction, echo cancellation, etc., in transmission

[0101] • digital / analog conversion, gain control, etc., on reception.

[0102] The figure here does not represent the acoustic treatments or the network (including treatments and equipment such as antennas, modems, etc.) connecting the two terminals, for the sake of conciseness and clarity.

[0103] In one embodiment, one of the devices A or B can be replaced by network equipment, which can be a "media gateway" (e.g.: IMS-AGW), a conference bridge (e.g.: MRFP), a voicemail server, etc. In order not to make the text too heavy, we describe below the example of a communication between two terminals A and B, with the understanding that one of the terminals can be a network termination (without sound recording / acoustic rendering in general).

[0104] In the case where one of the terminals is a network device, the interface providing the input signals of the encoder or recovering the output signals of the decoder / renderer is directly electrical.

[0105] Terminal A comprises a transmitter 101 and a receiver 103 while terminal B comprises a transmitter 151 and a receiver 153. The two terminals A and B are typically configured according to an exchange of offer(s) / response(s) according to the SDP protocol (for “Session Description Protocol” in English), via the respective blocks 100 and 150. Typically this SDP configuration is used at least during the instantiation (initialization or re-initialization) of the transmitter and the receiver; in variants of the invention it may be considered that this configuration is internal to the terminal and makes it possible to configure the SDP offer of the terminal and its response. In variants of the invention other signaling protocols for media capabilities / configurations (session descriptions) equivalent to or derived from SDP may be used in an equivalent manner.

[0106] Transmitters have coding functions, generating RTP packets by adding protocol headers to generate a transport format (payload format) corresponding to the coding format. A transmitter also has a function for possible adaptation of the transport (aggregation, redundancy or other) as well as a packet transmission function.

[0107] Thus, a transmitter of a communication device as described implements a method for generating an audio data transport format during a real-time communication session with another communication device.

[0108] For this purpose, as described in more detail later, the data transport format for an extended audio codec (IVAS) extending a mono audio codec (EVS) reuses the data transport format of said mono audio codec. At least one existing, unused or reserved field of the data transport format of said mono audio codec is used to signal the extended audio codec and / or to transmit at least information of the bit rate of the extended audio codec.

[0109] Receivers have functions of receiving RTP packets, extracting RTP header fields, decoding the transport format and decoding frames which may include jitter buffer management, audio decoding and correction of dropped frames.

[0110] Thus, a receiver of a communication device is able to receive and decode a transport format as generated by a transmitter of another communication device. To do this, it extracts at least one unused and existing field from the data transport format of the mono audio codec (EVS) and checks whether it signals the extended audio codec (IVAS) and / or gives information on the bit rate of the extended audio codec (IVAS). In an embodiment described later, it is another field of the data transport format which is used to transmit at least information on the bit rate of the extended audio codec.

[0111] Terminals A and B comprise a request decoding block (blocks 107 and 157), in particular for CMR requests as described in Figures 4 to 6 later. Block 107 or 157 consists of reading the relevant part of the packet associated with the codec (IVAS including EVS). It is assumed here that the result of the request decoding by blocks 107 and 157 is for example stored in a memory element called here "adaptation_info" containing a structure shared with the transmission part (blocks 101 and 151) and this "adaptation_info" element itself comprises a stored element "adaptation_info.updated" which is equal to 1 when a request (CMR or other) has been received and 0 otherwise. Block 101 or 151 checks, before encoding each new frame, whether the value of "adaptation_info.updated" is 1 and if so it configures the encoding according to the requested bit rate or the requested data format.

[0112] Several embodiments for generating the data transport format during communication between the two terminals A and B and the data transport format are now described. This data packet transport format exchanged between the terminals is compatible with immersive audio signal coding and decoding as implemented by an IVAS codec. This data transport format is also backward compatible with the coding and decoding of a mono audio signal, for example implemented by an EVS mono codec.

[0113] This data transport format can also be used for data storage and therefore be recorded in a memory accessible by communication terminals.

[0114] Before detailing a detailed example of a transport format for IVAS, we first look at the use cases of the IVAS codec.

[0115] The simplest use case for IVAS involves a call between two devices connected over a network, for example 5G. These devices can be smartphones equipped with multiple microphones and support the following formats:Mono: fallback to an EVS-compatible call, since EVS is included in IVASStereo with sound pickup on 2 phone microphones or with stereo sound from a microphone antenna processingBinaural: only in the case where a phone is connected to headsets that integrate microphones and thus capture natural binaural soundMASA: format with a mono or stereo signal supplemented by spatial information from a scene analysis as described in 3GPP Tdoc S4-220443, “MASA format updates,” April 2022, Source: Nokia, Orange

[0116] This list is not exhaustive, this example simply shows that the encoding format supported and displayed in SDP for IVAS would be for example a list:

[0117] [MONO, STEREO, BINAURAL, MASA1, MASA2]

[0118] MONO format can be considered redundant in this list and is necessarily included as a basic format in the IVAS codec since EVS is included in IVAS.

[0119] We can also consider that STEREO and BINAURAL are 2-channel formats that are coded almost identically, so they could be combined into a single type of coded format for defining terminal capabilities: the main justification for differentiating BINAURAL is to address use cases where microphones are placed on earpieces during communication, while STEREO is for microphones on a smartphone.

[0120] Similarly, MASA1 and MASA2 could both be supported, but in a call, a priori, only one format type should be used at a time, even if in theory the acoustic processing could be upstream of the MASA analysis and therefore allow real-time switching.

[0121] If both terminals A and B have the same coding / decoding / rendering capabilities, the terminal receiving such an offer could either choose a single format or keep a list (complete or partial) which supposes a potential format switching possible during the call. In reality it seems more realistic and simpler to select only a stereo or immersive format, with possibly the possibility of degrading the number of channels in the same type of format, in particular during transmission to maintain a coherent sound recording over time, decoupled from the coding.

[0122] In IVAS there is no internal signaling in the codec bitstream to differentiate STEREO and BINAURAL, so it is important to negotiate different formats, so that the rendering can adapt and the receiving terminal only accepts BINAURAL when headphones or earphones are used for direct listening.

[0123] Keeping more than one stereo / immersive format also raises an ambiguity related to the order of formats which normally indicates a preference (from best to worst).

[0124] Subsequently, it is assumed that a single stereo / immersive format is selected in SDP by the receiver of an IVAS SDP offer – this however allows the number of channels to potentially be reduced or increased (for example from HOA3 to FOA, or 5.1 to stereo and vice versa) within the limit of the maximum negotiated format (for example HOA3 or 5.1). The format selection rules remain to be defined if more than one format is offered and supported; they are beyond the scope of the invention.

[0125] If a change in encoding format is required, then SDP renegotiation must be employed.

[0126] This simple use case becomes more complicated if we imagine a scenario where one of the terminals switches from a 5G network (with IVAS support) to a 4G network that only supports the EVS codec. In this handover case, it is important not to drop the call and if possible to avoid end-to-end codec renegotiation during the call – which would involve a long interruption time – and also to avoid transcoding between IVAS and EVS in a media gateway in the network.

[0127] It should also be remembered that EVS is typically used with bit rates of 5.9 to 13.2 or 24.4 kbit / s in a "nb-wb" or "nb-swb" or "swb" type band, while IVAS bit rates only start from 13.2 kbit / s and the NB band is not supported by IVAS (except mono). It is therefore important that the IVAS payload format takes these constraints into account.

[0128] In the case of "handover", we could also imagine that some network solutions still prefer to transcode between IVAS and EVS in a "media gateway" (MGW) to avoid managing "handover" cases too finely; however, it would be possible, for example, that the MGW does not support all combinations of format / bit rate / coded band (e.g.: FB band) and the MGW could send a CMR to one of the terminals to force the encoder to change format / bit rate / band.

[0129] Another example of a use case is where an IVAS-compatible terminal (and therefore EVS since IVAS includes EVS) connects to a spatial conference bridge or a video game platform. Typically, the phone could only send in mono but receive, for example, in binaural or even in a multichannel format such as 5.1.

[0130] If the terminal receiving an immersive stream is connected to a headset or earphones, it seems preferable to receive or decode a binaural signal.

[0131] In any case the receiving terminal could indicate its decoding and rendering capabilities (renderer) in the form of a list such as with elements separated by hyphens to indicate that certain formats are included in others and indicated in the form of a minimum format-maximum format type interval (e.g.: STEREO-5_1), which would give a list of decoding / rendering capabilities like:

[0132] [MONO, BINAURAL, STEREO-5_1]

[0133] The receiving part generally depends on a rendering audio chain that must be initialized but can be decoupled from the decoding, which allows switching from headphones to listening on a single speaker of a smartphone and then a 5_1 system in a vehicle or a meeting room. Thus, unlike in the case of transmission, it is less problematic to signal a list of possible formats in reception. On the other hand, there may be some ambiguity on the order to use which could indicate a preference, but it can be assumed that the receiving terminal would use the most appropriate format depending on the rendering / listening context.

[0134] Here again, these negotiation aspects remain open and go beyond the scope of the invention. However, it is important to take into account the presence of SDP parameters that can impact the transport format. We assume here that the existing parameters for EVS are taken over in IVAS and that we add at least one additional parameter called for example "fmt-send" allowing to negotiate the coded stereo / immersive format capabilities. We can also consider that a terminal can also declare another parameter specific to IVAS "fmt-recv" which would indicate which reception formats are supported for rendering, which makes it possible to avoid selecting for example an MC714 coding if the rendering is limited to STEREO.

[0135] In the embodiments described below, transport formats of the immersive IVAS codec are specified that allow: to ensure continuity of service between the IVAS and EVS codecs to ensure a possible fallback to the mono format of the EVS codec provided in the IVAS codec, even if the current call has been negotiated for an immersive input / output format (stereo, etc.). Thus, it is possible to use the IVAS codec but to switch to the EVS codec at one end of the chain without stopping or renegotiating a call. to define a transport format of the IVAS codec that is as simple as possible as a natural extension of the transport format of the EVS codec.

[0136] Thus, according to the embodiments described below, the audio data transport format for an extended audio codec (IVAS) uses an extension of a data transport format of a mono audio codec (EVS), at least one existing field of the data transport format of the mono audio codec, unused or reserved, being used to convey at least information of the bit rate of the extended audio codec (stereo / immersive) or to signal the extended audio codec.

[0137] Before describing different embodiments according to the invention, examples are first given illustrating how the IVAS codec could be negotiated as an extension of the EVS codec at the SDP level. These examples are by no means definitive because the payload format is still completely open.

[0138] Annex 1 describes a (simplified) SDP offer from Terminal A

[0139] This first example deals with an SDP offering that allows initiating an audio call with the IVAS codec including the EVS codec. The UDP port, AVP profile and payload type (PT) are indicated in the media "m" line. The payload type numbers are PT=96 for IVAS or 97 for EVS.

[0140] The "rtpmap" and "fmtp" parts indicate the media format parameters.

[0141] It is assumed here that IVAS and EVS are offered in the package with the usual "telephone event" in telephony.

[0142] Appendix 2 describes a possible example of a (simplified) SDP response from terminal B.

[0143] Here we consider that only the IVAS offer is retained with restrictions and the “telephone event” is also retained (typically obligatory for telephone services).

[0144] The SDP parameters in Appendix 1 and 2 are described here for illustrative purposes only. For simplicity, it is assumed that the existing parameters (br, bw) are reused and extended, and parameters specifying the encoding and rendering formats supported by the terminals have been added (in the form of a list in square brackets).

[0145] Here we consider that the EVS SDP parameters corresponding to MONO are differentiated from the IVAS SDP parameters, in fact IVAS has particularities such as a bit rate starting at 13.2 kbit / s or an NB audio band which is not supported, which requires to differentiate in general these parameters. Rules can be defined to ensure consistency, for example on the "cmr" parameter.

[0146] In a first embodiment, the extension of the transport format is carried out by providing an extension of the header data format defining the EVS table of contents "ToC". Here, we are interested in the transport format of the EVS codec of the "header-full" mode.

[0147] According to this embodiment, we have as ToC byte the same format as EVS which is taken and extended for IVAS.

[0148] Thus the IVAS ToC byte has the same definition as the EVS ToC byte illustrated in and described above.

[0149] The first bit H=0 makes it possible to distinguish a ToC byte from a CMR byte in the payload header.

[0150] The F bit is set to 1 if the encoded frame corresponding to the ToC byte is followed by another frame. It is set to 0 if the corresponding frame is the last in the payload, which implies that the ToC byte is the last and no other ToC byte follows it.

[0151] In the first embodiment, the ToC byte for IVAS follows the structure of the ToC byte of EVS as shown in.

[0152] The FT_IVAS field (6 bits) is defined according to the tables shown in figures 3b, 3c, 3d.

[0153] In the, we see the same configuration as for the EVS codec in, that is to say that the FT field is divided into 3 parts EVS mode (1 bit), Unused / Q bit (1 bit) according to the EVS mode bit, and EVS bit rate (4 bits). However, here, the "Unused / Q bit" EVS field is replaced by a field specifying the use of an IVAS codec (bit at 1) or not (bit at 0). If this field is at 0, we remain in the definition of the table of the, on the other hand, if this bit is at 1, then the next field indicating the possible bit rates for EVS, will use the same codes to this time indicate the possible bit rates of the IVAS codec.

[0154] This illustrates this scenario where we see that the first 1-bit field, at 0, shows the EVS or IVAS mode (unlike the AMR WB IO mode of the), the second 1-bit field (at 1) shows that the IVAS codec is used and the codes of the following 4-bit field indicate the different possible bit rates of the IVAS codec (from 13.2 to 512 kbit / s, as well as IVAS SID and SPEECH_LOST). Here IVAS SID corresponds to the coded SID frames corresponding to the inactive frames when the IVAS DTX mode is activated; SPEECH_LOST indicates a lost frame.

[0155] The is identical to the for the EVS AMR-WB IO part.

[0156] Thus, the transport format from the EVS codec is extended for the IVAS codec, an EVS coded frame can be indicated as in the EVS payload format and there is no need to provide another format specific to the IVAS codec because all the data necessary to indicate the type of IVAS frame can be represented by exploiting the unused bit of the FT field in the EVS payload format.

[0157] Alternatively, additional extensions to the ToC byte can be provided. These are not described here as they are still under discussion and to be defined for the IVAS codec, but they can be similar for example to the "PI_frame" idea of ​​the second proposal cited in the state of the art. Since there is no field available in the, these extensions could be signaled in different ways. Either using an RTP header extension (outside the payload) or taking another code like the 'For future use' code of the which is normally for EVS Primary (EVS bit rate = '1101') but which in the IVAS context would indicate an IVAS-specific data frame.

[0158] In a second embodiment, at least one existing codec matching request (CMR) byte, comprising an unused or reserved field of the EVS mono audio codec, is used to transmit at least information on the bit rate of the codec to be used, and optionally, according to the invention, an additional byte specific to IVAS can be added.

[0159] Here we are interested in the transport format of the EVS codec in header-full mode. To define the transport format of the IVAS codec, we extend the transport format of the EVS codec by extending the use of the CMR byte.

[0160] Thus the IVAS CMR byte according to the invention follows the same definition as the EVS CMR byte illustrated in and reproduced here in the. described above with a minor extension of the T and D fields; the latter is here named D_IVAS to differentiate it from the D field defined in EVS.

[0161] The first bit H has the value H=1 to specify that a CMR request exists.

[0162] The fields T (3 bits) and D_IVAS (4 bits) are defined as in the tables in the, except for the values ​​associated with the field T=' 111' which are modified according to the values ​​defined in the table illustrated in the.

[0163] Thus, the values ​​defined as 'Reserved' in the EVS CMR format are now defined as IVAS codec bitrates from 13.2 to 512 kbit / s. This embodiment therefore makes it possible to define a bitrate adaptation request, i.e. the bitrate that the terminal sending the CMR request wishes to receive.

[0164] In a third embodiment, it will be possible to allow – unlike EVS – to have more than one CMR byte in a packet for the IVAS codec, as illustrated in . If a CMR must be transported in the current packet, a first byte can first be inserted in the payload header in the second embodiment and as in , which allows a request to change the IVAS bit rate or a switch to a mono EVS mode to be made - unless NO_REQ is used, in which case the CMR carries an "empty" request. Then, optionally, a second CMR byte can be added, whose first most significant bit H will also be set to 1, to differentiate the byte from a ToC byte. The remaining 7 bits then represent a change of format on transmission, for example we can use the last 5 FMT_IVAS bits to indicate the coding format as in and the other 2 remaining BW_IVAS bits to indicate the audio band (WB, SWB, FB for IVAS) as in.This optional addition of a second CMR byte does not pose any ambiguity problem for writing / reading the payload: when reading, we start by reading the first byte, if the MSB bit is at 1, it is a CMR, then we read the second byte, if the MSB is also at 1, it is still a CMR request, and in this case the third byte will be of type ToC with an MSB at 0 according to the invention.

[0165] Note that it includes reserved fields to allow possible extension of IVAS coding formats, for example for applications with separate rendering between network (edge, cloud) and terminal using the split rendering method.

[0166] In a fourth embodiment of the invention, the values ​​associated with the field T=' 111' are modified in another way, as illustrated in. In this embodiment, an existing unused or reserved codec adaptation request field of the EVS mono audio codec is used to inform of the existence of a second code specifying the bit rate of the codec to be used in reception. Thus, in the example illustrated in, the codes D='0000' and D='0001' initially defined as "Reserved" for the EVS codec, are replaced by other codes named "Escape Input" and "Escape Output", where 'escape' is here an indication of extension of a CMR byte by another byte.

[0167] Thus, if the CMR byte is defined by T = '111' and D='0000', in the case of "Escape Input", this indicates that an additional CMR byte is defined immediately after the first CMR byte and this second byte has values ​​for example according to the format CF (4 bits) BR (4 bits) as defined in the table shown in. This additional byte indicates a request to change the immersive coding format (values ​​defined by the code CF for "code format") and / or the bit rate (values ​​defined by the code BR for "Bit rate") in reception, i.e. the bit rate that the terminal sending the CMR request wishes to receive.

[0168] Unlike the third embodiment, this variant with reference to the one does not allow to differentiate STEREO / BINAURAL nor MASA1 / MASA2 which could be a problem in certain applications.

[0169] Similarly, if the CMR byte is defined is T = '111' and D='0001', in the case of "Escape Output", this indicates that an additional CMR byte is defined immediately after the first CMR byte and this second byte has values ​​for example according to the format CF (4 bits) BR (4 bits) as defined in the table shown in. This additional byte indicates a request to inform the decoding / immersive rendering format (values ​​defined by the code CF for "code format") and / or request to change the bit rate (values ​​defined by the code BR for "Bit rate") in transmission, i.e. the bit rate that the sender of the CMR request sends.

[0170] In a fifth embodiment, it will be possible to combine previous variants to have a CMR byte format as illustrated in.

[0171] In variants, the CMR field(s) (in one or two CMR bytes) according to the invention may be inserted into an RTCP packet (of the RTCP-APP or RTCP-FB type) in the case where an adaptation according to the AVPF profile is authorized; in this case the CMR field(s) may not be used in the RTP packets. In addition, the use of the CMR (only change of bit rate and / or format) may be controlled by an SDP parameter such as the “cmr” parameter already defined in the payload format of the EVS codec.

[0172] The invention has been described so far with a transport format in "header-full" mode. In variants of the invention, a default transport format mode as in EVS can be used for the IVAS extended codec. There are in fact two ways to apply this default mode: EVS compact mode reused as is (with protected sizes as defined in 3GPP TS 26.445 Annex A) and header-full mode for EVS / IVAS according to the invention (with zero padding to avoid size conflicts). IVAS compact mode for bit rates from 13200 to 512000 bit / s with protected sizes as in Table 3 below.

[0173] ModePayload Size (bits)Bitrate (kbps)IVAS SID1045.2IVAS26413.2IVAS32816.4IVAS48824.4IVAS64032IVAS96048IVAS128064IVAS192096IVAS2560128IVAS5120256IVAS7680384IVAS10240512

[0174] In variants, the Extended Codec Transport Format (IVAS) may also include the ability to extend the RTP header, for example to carry information such as a terminal's orientation.

[0175] The payload format of the IVAS codec according to the invention assumes that a format change is possible at least between mono and stereo / immersive format, both on transmission and reception.

[0176] Mono (EVS) and stereo / immersive (IVAS) switching must be able to be done at any time frame, in particular following the reception of a CMR according to the invention.

[0177] We remind you that the algorithmic delay of the EVS codec is 30.9375 ms in NB and 32 ms for the other bands (WB, SWB, FB).

[0178] For IVAS, the algorithmic delay is 32 ms (e.g. stereo) to 38 ms. To be able to switch without time alignment problems, it is therefore important to synchronize the encoding and decoding / rendering parts by applying the necessary delay difference depending on the operating mode. It should also be noted that in the case of binaural rendering, a possible delay due to binauralization and the possible presence of additional reverberation must be taken into account to ensure switching without artifacts.

[0179] It is assumed here that switching can be done by "resetting" the coding. In order to avoid too long an audio signal convergence time, it is possible to resynchronize memories / states of the EVS / IVAS codec according to the past frames and to use an "intelligent" reduction or increase of channels / parameters depending on the formats.

[0180] When decoding, it is also necessary to ensure that a frame loss does not severely impact switching (e.g., first mono frame lost after a series of IVAS frames, before receiving the other EVS frames).

[0181] Here we take the concrete case of switching between IVAS STEREO and EVS (mono).

[0182] At frame T we assume that we are in STEREO, at frame T+1, following the reception of a CMR, the coding switches to EVS mono mode. Ideally, the audio bands coded for IVAS and EVS should be identical (e.g. SWB). Since IVAS includes a channel reduction mode from stereo to mono compatible with EVS, the main difficulty in switching is to avoid a discontinuity of the signal when it will be decoded and also to properly initialize the codec states during switching.

[0183] At decoding (at reception), the STEREO to MONO switching is detected when reading the payload which is STEREO at frame T and MONO (EVS) at frame T+1. In this case, the "reset" is a simple solution which could still result in audio artifacts. Here we can consider that the algorithmic delay at decoding can be exploited to crossfade the audio outputs of frames T and T+1.

[0184] The same approach can be used for the other switching direction (from MONO to STEREO) with a reset of the states and the use of a crossfade during decoding.

[0185] This example generalizes to other stereo formats: for the BINAURAL format, the processing is very similar to the STEREO case. For other immersive formats, switching to MONO involves reducing the number of channels. Switching can be achieved by resetting the codec states, possibly using a crossfade.

[0186] Now describes an example of hardware embodiment of a communication device, for example a communication terminal implementing the method for generating a data transport format according to the invention, as described with reference to Figures 3 to 6 described above.

[0187] The terminal TA comprises a storage space 11, for example a memory MEM, a processing unit 10 comprising a processor P, controlled by a computer program PG, stored in the memory 11 and implementing the transport method according to the invention.

[0188] Upon initialization, the code instructions of the program PG are for example loaded into a RAM memory (not shown) before being executed by the processor P of the processing unit 10. The program instructions can be stored on a storage medium such as a flash memory, a hard disk or any other non-transitory storage medium.

[0189] The processing processor implements the transport method as described with reference to, and according to the formats described with reference to Figures 3 to 6, according to the program instructions PG.

[0190] The terminal TA comprises a communication module 12 capable of receiving and transmitting real-time data to a communication network R and capable of reading signaling or configuration data of the SDP type 13 either through the signaling transmitted in the network or in the memory 11 of the terminal.

[0191] The terminal comprises a receiver module 14 capable of receiving and decoding a transport format as generated by a transmitter of another communication device and as described with reference to Figures 3 to 6. For this purpose, it extracts fields from the RTP header, in particular the values ​​transmitted in the existing, unused or reserved fields, of the data transport format of the mono audio codec, in order to know at least the bit rate information of the extended audio codec. The receiver module is also capable of managing a reception memory, decoding the received frames and correcting any frame losses.

[0192] The terminal comprises a transmitter module 15 capable of encoding and generating RTP packets by adding the protocol headers to generate a transport format (payload format) corresponding to the coding format. A transmitter also has a function for possible adaptation of the transport (aggregation, redundancy or other) as well as a packet transmission function.

[0193] Thus, the transmitter module implements a method for generating an audio data transport format during a real-time communication session with another communication device.

[0194] For this purpose, the data transport format for an extended audio codec (IVAS) extending a mono audio codec (EVS) reuses the data transport format of the mono audio codec. At least one existing, unused or reserved field of the data transport format of said mono audio codec is used to signal the extended audio codec and / or to transmit at least information of the bit rate of the extended audio codec. Several embodiments of this data transport format are described with reference to Figures 3 to 6.

[0195] The term module can correspond to a software component as well as to a hardware component or a set of hardware and software components, a software component itself corresponding to one or more computer programs or subroutines or more generally to any element of a program capable of implementing a function or a set of functions as described for the modules concerned. In the same way, a hardware component corresponds to any element of a hardware assembly capable of implementing a function or a set of functions for the module concerned (integrated circuit, smart card, memory card, etc.)

[0196] The terminal is, for example, a telephone, a smartphone, a tablet, a computer, a residential gateway or a connected object. APPENDIX 1

[0197] Offre SDP (simplifiée) du terminal Am=audio 54590 RTP / AVP 96 97 98a=rtpmap:96 IVAS / 16000a=fmtp:96 evs=[br=9.6-24.4;bw=swb;max-red=0]; ivas=[br=13.2-48;bw=swb;fmt-send=[ STEREO,MASA1-MASA2];fmt-recv=[STEREO,BINAURAL]]a=rtpmap:97 EVS / 16000a=fmtp:97 br=9.6-24.4;bw=swb;max-red=0a=rtpmap:98 telephone-event / 16000a=fmtp:98 0-15a=ptime:20a=maxptime:20 ANNEXE 2

[0198] Réponse SDP (simplifiée) du terminal Bm=audio 54590 RTP / AVP 96 98a=rtpmap:96 IVAS / 16000a=fmtp:96 evs=[br=9.6-24.4;bw=swb;max-red=0]; ivas=[br=13.2-24.4;bw=swb;fmt-send=[ STEREO];fmt-recv=[STEREO]]a=rtpmap:98 telephone-event / 16000a=fmtp:98 0-15a=ptime:20a=maxptime:20

Claims

Method for generating an audio data transport format during a real-time communication session between communication devices (A, B), in which the data transport format for an extended audio codec (IVAS), extending a mono audio codec (EVS), reuses the data transport format of said mono audio codec, and where at least one existing, unused or reserved field of the data transport format of said mono audio codec is used to signal the extended audio codec and / or to transmit at least information of the bit rate of the extended audio codec. The method of claim 1, wherein an unused and existing field of the data transport format of said mono audio codec is used to signal the extended audio codec and another field is used to convey at least information of the bit rate of the extended audio codec. The method of claim 1, wherein at least one existing, unused or reserved codec matching request (CMR) field of the mono audio codec is used to convey at least information of the bit rate of the extended codec to be used. The method of claim 3, wherein each of the possible bit rates of the extended codec is indicated in an existing, unused or reserved codec adaptation request field of the mono codec. The method of claim 3, wherein an existing, unused or reserved, codec adaptation request field of the mono audio codec is used to inform of the existence of a second code specifying a format and / or bit rate of the extended codec to be used in reception. The method of claim 3, wherein an existing, unused or reserved, codec adaptation request field of the mono audio codec is used to inform of the existence of a second code specifying a format and / or bit rate of the extended codec used in transmission. Communication terminal comprising a processing circuit capable of implementing a method for generating an audio data transport format during a real-time communication session with a communication device, in which the data transport format for an extended audio codec (IVAS), extending a mono audio codec (EVS), reuses the data transport format of said mono audio codec, at least one existing, unused or reserved field of the data transport format of said mono audio codec being used to transmit at least information on the bit rate of the extended audio codec. Audio data transport format for an extended audio codec (IVAS), extending a mono audio codec, reusing the data transport format of said mono audio codec (EVS) in which at least one existing, unused or reserved field of the data transport format of said mono audio codec is used to signal the extended audio codec and / or to transmit at least information of the bit rate of the extended audio codec. Computer program comprising code instructions for implementing the steps of the generation method according to one of claims 1 to 6, when these instructions are executed by a processor. Storage medium, readable by a processor, storing a computer program comprising instructions for executing the generation method according to one of claims 1 to 6.

Images