Improvements to wireless audio transmission

Abstract

An audio encoder and computer readable medium with instruction to execute a method of encoding 110 an audio signal that minimises audio degradation and the audible impact of missing audio data packets

Description

Field of the Invention The present invention relates to methods to minimise the audible effect of missing packets in wireless transmission of audio. Background to the Invention Digital audio is typically transmitted wirelessly by an audio encoder encoding the audio in sections into data packets, where the resultant data packets are then transmitted wirelessly to receivers. The received data packets are then decoded by a corresponding audio decoder. For various reasons the wireless transmission of data packets is not perfect. Transmitters have various strategies to reduce the likelihood of packets not being received, such as forward error correction or acknowledgement and retransmission but still some data packets do not get received successfully, especially if it’s desired to minimise the signal delay imposed by the data transmission. Consequently the audio decoder must be able to cope with the possibility that some data packets are missing, and missing packet concealment algorithms are used to invent suitable filler audio to conceal the gap described by a missing data packet. Such filler audio is not a perfect substitute for the intended audio and there is a need to reduce the damage to the decoded audio caused by missing packets. Summary of the Invention According to a first aspect of the present invention, there is provided a method for encoding audio to a sequence of data packets, the method comprising also furnishing an importance value for each data packet which indicates the degree of audible degradation that would result if data packet were not provided to the corresponding decoding method. In this way, a subsequent wireless transmitter can make better decisions when dealing with problems that might cause packet loss since it is aware of the relative importance of each data packet to the final decoded audio. Whilst the encoding method could explicitly model the packet loss concealment resulting from missing packets, preferably it calculates the importance value from values or statistics it has computed in the process of encoding the data packet. In this way, the importance value can be furnished with negligible extra computational cost over the task of encoding. An example of such a statistic is whether a transient was detected in encoding the data packet. If it was, the packet has high importance value because a missing packet concealment algorithm is unlikely to correctly guess the presence of the transient. Another example of such a statistic is the extent to which an innovation signal (the difference between the audio and a prediction of the audio) varies in variance during the packet. A packet concealment algorithm is unlikely to be able to satisfactorily model innovation of varying variance so such a packet has high importance. If an audio codec were to continuously update a model of some characteristic of audio (such as a prediction filter modelling the audio’s spectral shape), then another example of such a statistic might be the degree to which that model changes as the data packet is encoded. Another way of using statistics produced in the process of encoding the packet is to compare them to corresponding values produced from encoding prior packets. An example of these second sort of statistics might be a measurement of spectral power in several frequency bands, as a packet concealment algorithm would typically assume the power in each frequency band was similar to previous packets. Another example would be the loudness of the audio. If this packet is louder than the general loudness level across the previous second or so then it should have a higher importance value since any errors in its reproduction will be made more apparent to the listener due to its loudness. The reverse applies if it is quieter. According to a second aspect of the invention, the importance value is coupled to a wireless transmitter which transmits the data packets to one or more wireless receivers. The wireless transmitter does not transmit the importance value, but uses it to make better decisions affecting packet loss. Preferably these decisions apply more resources to ensuring the successful delivery of data packets with higher importance values than applied to packets with lower importance values. In this way the wireless transmitter uses the information to maximise the probability of successful reception of the important packets, and so reduces the audible consequences of whatever packet loss still occurs since it is the lower importance packets which are more susceptible. The transmitter might do this by applying greater forward error correction to packets with higher importance values. Or the transmitter may sometimes find that successful delivery of all packets is infeasible and it has to make a choice about which packet it discards. It would naturally discard the packet with lower importance value. An audio encoder may use the above methods, or a computer program stored on a computer readable medium describe how to implement them. As will be appreciated by those skilled in the art, the present invention is capable of various implementations according to the application, as will be apparent from the following discussion. Brief Description of the Drawings Embodiments of the invention will now be described by way of example with reference to the accompanying figures in which: Figure 1 shows an audio encoder encoding segments of audio to data packets for wireless transmission. Additionally the effect of missing packet concealment is modelled and the resultant audio degradation quantified and used to guide the wireless transmitter’s choices. Figure 2 shows how useful statistics can be extracted from the audio encoder. An importance value can be calculated from these and how they differ from preceding packets and used to guide the wireless transmitter’s choices. Figure 3 presents as a flowchart specific information that might be extracted from the audio encoding process and used to evaluate the importance of reliable delivery of this specific packet. Figure 4 shows 20ms of example audio waveform subdivided into 5ms packets. Detailed Description Wireless data transmission suffers from problems such as interfering signals, multipath interference or extreme range. These cause data corruption and unreliable reception of data packets. There are techniques to mitigate these reliability issues, such as forward error correction, or error detection, packet acknowledgement and retransmission of failed packets. These mitigation techniques use limited resources, such as link capacity or latency, and so there are limitations on the extent to which they can ensure all transmitted data packets are successfully received. Consequently an audio codec for use with wireless data transmission needs to be able to cope with missing data packets and missing packet concealment algorithms attempt to concoct suitable audio to cover and conceal the gap corresponding to a missing packet. The resulting audio will be poorer quality than if the decoder had been able to decode the true packet. How much poorer depends on the quality of the packet loss concealment algorithm. But it also depends on the nature of the audio conveyed by the missing packet. Often, audio has a consistent level and spectrum from one packet to another. In those circumstances, packet loss concealment can do a good job of inventing plausible filler audio and so the audible impact of the missing packet is benign. But sometimes, the audio in a packet contains transient sounds, note onsets and other events that are not obvious from analysing the preceding audio and in these circumstances the resulting audio is less plausible and the audible impact of the missing packet worse. Figure 1 shows audio (101) being encoded by an encoder (110) into data packets (102) for wireless transmission by a transmitter (111). The audio may comprise multiple channels and each data packet corresponds to a section of the audio (sections may overlap due to windowing). For each data packet, we model the effect of its failure to arrive at a decoder by executing a missing packet concealment algorithm (120). This uses preceding audio (or perhaps information internal to the codec from processing previous packets) to produce estimated filler audio. We want to quantify how satisfactory that filler audio is as a substitute for the actual section of audio. Figure 1 illustrates this by comparing it with the genuine audio in a masking model (121) and measuring (122) how perceptually different they are. We call the resulting value (130) an importance value as it quantifies how important delivery of that packet is to satisfactory audio decode. That importance value (130) is then fed into the transmitter (111) so that when it makes trade-offs which affect the reliability of packet delivery, it can prioritise the more important packets at the expense of the less important packets. Figure 1 illustrates the concept, but it is needlessly expensive to implement. Useful importance values can be far more easily computed from internal signals within the audio encoder than by going to the trouble of generating actual concealment audio and analysing it. However there are a variety of successful approaches to audio coding, which differ in what internal information is computed and details of how importance values are computed will likewise differ. Figure 2 illustrates this approach. Segments of audio are encoded into packets of data (102) for wireless transmission. Various statistics (201) produced in the process of encoding the segment of audio are extracted (210) for calculation (211) of the data packet’s importance value (130). Statistics that indicate the audio contains a transient or is changing would give rise to a high importance value as these features suggest that prior audio may be a poor guide to current audio. These statistics (201) are also compared (213) with delayed (212) statistics from encoding prior packets and the extent to which they have changed is also used in calculating (211) the data packet’s importance value. Statistics that differ greatly from those in prior packets would give rise to a high importance value as they suggest the packet’s audio differs from its predecessors. The resultant importance value (130) is then used to guide the transmitter as it makes trade-offs that affect data reliability. Figure 3 is a flowchart showing criteria that could be used to quantify the importance value. The criteria are neither essential nor exhaustive; not all criteria are applicable to all codecs and further criteria could be added. As a general description, missing packet concealment algorithms will produce a smooth continuation of the preceding audio, maintaining a similar level and spectrum. A packet should have a high importance value to the extent that the audio it describes is surprising and not a natural mechanical continuation of the preceding audio. The flowchart shows various ways in which surprising characteristics of the audio might be identified and used to increase the importance value for the packet. Is the spectral shape similar to recent packets in each channel (301)? This might be evaluated by looking at the spectral power in a variety of frequency bands (many codecs will calculate these statistics for their own internal operations) and comparing with corresponding statistics for previous packets. If the spectral shape has changed significantly, then the importance value should be larger. Some audio codecs characterise packets as having a transient nature or not (in order to adjust the tradeoff between time and frequency resolution in a transformation). Transients are typically unexpected and not likely to be reproduced by a missing packet concealment algorithm, so the encoder analysis (302) which decides how transient a packet is will have direct application to the importance value. Some other audio codecs might implement linear predictive coding, subtracting a predicted signal value from the input audio to produce an innovation signal. A large variation in innovation variance would be indicative of the packet having a transient nature. Some audio codecs continuously adapt an implicit model of the audio. For example an ADPCM codec might continuously adapt a prediction filter and maintain an estimate of the level of innovation signal resulting from subtracting predicted audio from the input audio. It would do this on each channel and possibly in multiple spectral bands. If this model is similar after encoding the packet to before, then it’s reasonable to conclude that the packet was unimportant and missing packet concealment would be effective, if the model has changed significantly (303) then the packet was important. All of these considerations want to be put into the context of whether the packet’s audio is loud or quiet (304) compared to the typical prevailing level. Any defects in missing packet concealment audio will be far more objectionable in a loud section than a quiet section. Having considered various factors and produced an importance value, it can be associated (305) with the data packet to inform the transmitter’s choices. The resultant importance value is not a figure that needs accurate calculation to high accuracy. Most of the benefit accrues from identifying those few packets whose loss would be particularly damaging to the decoded audio, and most codecs will have internal calculated values that identify them readily available without requiring much additional computation. Figure 4 shows a section of waveform (recorded from a xylophone), divided up into 5ms sections of audio, each corresponding to a data packet. The first packet, P0, describes a reverberation tail from the previous strike. It is sinusoidal and a missing packet concealment algorithm will perform very well on this packet. It should be associated with a low importance value. The second packet, P1, describes a note onset. A missing packet concealment algorithm will perform poorly on this packet, since nothing about the preceding audio suggests that this note onset was about to happen. It should be associated with a high importance value. These importance values communicate to the wireless transmitter that it should make every effort to successfully communicate P1 to the receiver. If its efforts to ensure successful receipt of P1 make it less likely that PO is successfully received then that is a price worth paying. The remaining packets P2 and P3 show the early evolution of the note. Each differs in character from its predecessor as the note evolves, so a missing packet concealment algorithm will face more difficulty than it did with the extremely predictable PO. But the difference is far less marked than it was for P1, so these packets might be associated with medium importance values. The wireless transmitter should prioritise the delivery of P1 over P2 or P3 but not to the same extent as over PO. How should the transmitter utilise the importance value? It will vary, depending on how it deals with transmission errors. As a general statement, it can benefit from the information by devoting more resources to ensuring the reliable delivery of high importance packets and may balance this by devoting fewer resources to ensuring the reliable delivery of low importance packets. If the transmitter uses forward error correction, it may apply heavier error correction to packets with a high importance value and lighter error correction to packets with a low importance value. If the transmitter uses acknowledgements and retransmissions, then sometimes there may be insufficient airtime to retransmit all the failed packets. With multiple packets inflight, it can choose to discard the packet(s) with lower importance values. Higher importance packets that are approaching expiry may be retransmitted on multiple channels to increase the chance of successful receipt.

Claims

1. A method for encoding audio comprising one or more channels to a sequence of data packets each corresponding to a section of the audio wherein:for each data packet, an importance value is also furnished indicating how much audible degradation would occur if the data packet were not provided to a corresponding decoding method.

2. A method according to claim 1 wherein the importance value for the data packet is calculated in dependence on first statistics computed in the process of encoding the data packet.

3. A method according to claim 2 wherein the first statistics comprise information relating to whether the section of audio contains a transient.

4. A method according to claim 2 or 3 wherein the first statistics comprise the variation in innovation variance across encoding the data packet.

5. A method according to claims 2 to 4 wherein the first statistics comprise the degree of change in modelled audio characteristics whilst encoding the data packet.

6. A method according to claims 1 to 5 wherein the importance value for the data packet is calculated in dependence on a comparison between second statistics calculated from the audio in encoding this data packet and corresponding second statistics calculated in encoding prior data packets.

7. A method according to claim 6 wherein the second statistics comprise the spectral power measured in several frequency bands.

8. A method according to claim 6 or 7 wherein the second statistics comprise a measure of the audio loudness.

9. A method according to claims 1 to 8 wherein the data packets are transmitted by a wireless transmitter to one or more wireless receivers and the importance value is coupled to the wireless transmitter but not transmitted to the receivers.

10. A method according to claim 9 wherein the wireless transmitter applies more resources to ensuring the successful delivery of data packets with higher importance values than data packets with lower importance values.

11. A method according to claim 10 wherein the wireless transmitter applies a higher level of forward error correction to data packets with higher importance values.

12. A method according to claim 9 or 10 wherein when the wireless transmitter finds it cannot successfully deliver all the data packets and has a choice about which data packet to discard, it discards a data packet with lower importance value to prioritise the delivery of data packets with higher importance value.

13. An audio encoder also furnishing an importance value using the method of any of claims 1 to 12.

14. A computer readable medium comprising instructions that, when executed by one or more processors, cause said one or more processors to perform the method of any of claims 1 to 12.

Images