Packet loss compensation device and packet loss compensation method, and voice processing system

Abstract

To provide a packet loss concealment apparatus and a packet loss concealment method, wherein spatial artifacts such as incorrect angle and diffuseness may be avoided as far as possible, and to provide an audio processing system.SOLUTION: According to an embodiment, the packet loss concealment apparatus is provided for concealing packet losses in a stream of audio packets, each audio packet comprising at least one audio frame in transmission format comprising at least one monaural component and at least one spatial component. The packet loss concealment apparatus may comprise a first concealment unit for creating the at least one monaural component for a lost frame in a lost packet and a second concealment unit for creating the at least one spatial component for the lost frame.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] This specification generally relates to audio signal processing. Embodiments of this specification relate to compensating for artifacts resulting from spatial audio packet loss that occurs during the audio transmission process over a packet-switched network. More specifically, embodiments of this specification relate to a packet loss compensation apparatus, a packet loss compensation method, and an audio processing system including the packet loss compensation apparatus.

Background Art

[0002] Voice communications can be subject to various quality issues. For example, when voice communications are performed over a packet-switched network, some packets may be lost due to delay jitter within the network, or due to adverse channel conditions such as fading or Wi-Fi interference. Lost packets result in clicks, pops, or other artifacts, which significantly degrade the quality of speech perceived by the receiver. To counteract the undesirable effects of packet loss, packet loss concealment (PLC) algorithms, also known as frame erase compensation algorithms, have been proposed. Such algorithms typically operate at the receiver side by generating a synthesized voice signal to cover the lost data (erased portions) in the received bitstream. These algorithms are primarily proposed for monaural signals in either the time domain or the frequency domain. Based on whether compensation occurs before or after decoding, PLC for monaural channels can be classified into coding domain, decoding domain, or a hybrid domain. Applying PLC for monaural channels directly to multichannel signals may result in undesirable artifacts. For example, after decoding each channel, PLC (Programmable Logic Controller) of the decoded region may be performed separately for each channel. One drawback of this method is that, because it does not consider the correlation between channels, not only spatially distorted artifacts but also unstable signal levels may be observed. Spatial artifacts such as inaccurate angles and diffusion can significantly degrade the perceptual quality of spatial speech. Therefore, there is a need for PLC algorithms for speech signals that encode multi-channel spatial fields or sound fields. [Overview of the project] [Means for solving the problem]

[0003] According to one embodiment of this specification, a packet loss compensation device is provided for compensating for packet loss in a stream of voice packets, wherein each voice packet includes at least one voice frame in a transmission format that includes at least one monaural component and at least one spatial component. The packet loss compensation device comprises a first compensation unit for creating at least one monaural component for the lost frame in the lost packet, and a second compensation unit for creating at least one spatial component for the lost frame.

[0004] The packet loss compensation device described above may be applied to an intermediate device such as a server, for example, an audio conferencing mixing server, or a communication terminal used by an end user. This specification also provides a voice processing system comprising a server equipped with the above-described packet loss compensation device and / or a communication terminal equipped with the above-described packet loss compensation device.

[0005] Another embodiment of this specification provides a packet loss compensation method for compensating for packet loss in a stream of voice packets, wherein each voice packet comprises at least one voice frame in a transmission format comprising at least one monaural component and at least one spatial component. The packet loss compensation method comprises creating at least one monaural component for the lost frame in the lost packet, and / or creating at least one spatial component for that lost frame.

[0006] This specification also provides a computer-readable medium on which computer program instructions are recorded, and which, when executed by a processor, allows the processor to perform the packet loss compensation method described above.

[0007] This specification is illustrated by the accompanying drawings, not by any limit, and in the drawings, the same reference numerals refer to similar elements. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram illustrating an exemplary voice communication system to which the embodiments of this specification can be applied. [Figure 2] This is a schematic diagram showing another exemplary voice communication system to which the embodiments of this specification can be applied. [Figure 3] This figure shows a packet loss compensation device according to one embodiment of this specification. [Figure 4] This figure shows a specific example of a packet loss compensation device. [Figure 5] This figure shows the first compensation unit 400 of Figure 3, according to one modified example of the embodiment shown in Figure 3. [Figure 6] This figure shows a modified example of the packet loss compensation device shown in Figure 5. [Figure 7] This figure shows the first compensation unit 400 of Figure 3, according to another modification of the embodiment shown in Figure 3. [Figure 8] This figure shows the principle of the modified example shown in Figure 7. [Figure 9A] This figure shows the first compensation unit 400 of Figure 3, which is a further modified example of the embodiment of Figure 3. [Figure 9B] This figure shows the first compensation unit 400 of Figure 3, which is a further modified example of the embodiment of Figure 3. [Figure 10] This figure shows a specific example of a modified version of the packet loss compensation device shown in Figure 9A. [Figure 11] This figure shows a second converter in a communication terminal according to another embodiment of this specification. [Figure 12] This figure shows the application of a packet loss compensation device according to the embodiment of this specification. [Figure 13] This figure shows the application of a packet loss compensation device according to the embodiment of this specification. [Figure 14] This figure shows the application of a packet loss compensation device according to the embodiment of this specification. [Figure 15] This is a block diagram illustrating an exemplary system for carrying out the embodiments of this specification. [Figure 16]It is a flowchart showing compensation for monaural components in the packet loss compensation method according to embodiments and modifications of this specification. [Figure 17] It is a flowchart showing compensation for monaural components in the packet loss compensation method according to embodiments and modifications of this specification. [Figure 18] It is a flowchart showing compensation for monaural components in the packet loss compensation method according to embodiments and modifications of this specification. [Figure 19] It is a flowchart showing compensation for monaural components in the packet loss compensation method according to embodiments and modifications of this specification. [Figure 20] It is a flowchart showing compensation for monaural components in the packet loss compensation method according to embodiments and modifications of this specification. [Figure 21] It is a flowchart showing compensation for monaural components in the packet loss compensation method according to embodiments and modifications of this specification. [Figure 22] It is a block diagram of an exemplary sound field encoding system. [Figure 23A] It is a block diagram of an exemplary sound field encoder. [Figure 23B] It is a block diagram of an exemplary sound field decoder. [Figure 24A] It is a flowchart of an exemplary method for encoding a sound field signal. [Figure 24B] It is a flowchart of an exemplary method for decoding a sound field signal.

Best Mode for Carrying Out the Invention

[0009] Embodiments of this specification will be described below with reference to the drawings. Note that, for clarity, expressions and descriptions regarding elements and processes that are known to those skilled in the art but not necessary for understanding this specification are omitted in the drawings and the description.

[0010] As will be understood by those skilled in the art, embodiments of this specification may be embodied as systems, devices (e.g., mobile phones, portable media players, personal computers, servers, television set-top boxes, or digital video recorders, or any other media players), methods, or computer program products. Accordingly, embodiments of this specification may be in the form of hardware embodiments, software embodiments (e.g., firmware, resident software, microcode), or embodiments combining both software and hardware embodiments, all of which may be collectively referred to herein as “circuits,” “modules,” or “systems.” Furthermore, embodiments of this specification may be in the form of computer program products embedded in one or more computer-readable media, the computer-readable media including computer-readable program code embedded therein.

[0011] Any combination of one or more computer-readable media may be used. Computer-readable media may be computer-readable signal media or computer-readable storage media. Computer-readable storage media may, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any appropriate combination thereof. Further specific examples of computer-readable storage media (non-exclusive enumeration) would include: electrical connections including one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any appropriate combination thereof. In the context of this specification, computer-readable storage media may be any tangible media that contains or can store programs for use by, or in connection with, a system, apparatus, or device that executes instructions.

[0012] A computer-readable signaling medium may include data signals propagated along with computer-readable program code embedded within the medium, for example, within the baseband or as part of a carrier wave. These propagated signals can take various forms, including, but are not limited to, electromagnetic signals, optical signals, or any appropriate combination thereof.

[0013] A computer-readable signaling medium may be any computer-readable medium other than a computer-readable storage medium that can communicate, propagate, or transmit programs for use by, or in connection with, a system, apparatus, or device that executes instructions.

[0014] Program code embedded in a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless cables, wired cables, fiber optic cables, RF, or any suitable combination thereof.

[0015] Computer program code for performing the actions described herein may be written in any combination of one or more programming languages, such as object-oriented programming languages ​​such as Java®, Smalltalk, and C++, and traditional procedural programming languages ​​such as the C programming language and similar languages. The program code may be executed as a standalone software package on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the context of the last example, the remote computer may be connected to the user's computer via any type of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (for example, via the Internet using an Internet service provider).

[0016] Aspects of this specification will be described below with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of this specification. It will be understood that each block in a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, are executable by computer program instructions. These computer program instructions may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device for manufacturing machines, and as a result, instructions executed via the computer processor or other programmable data processing device create means for performing the functions / operations specified in one or more blocks of the flowchart and / or block diagram.

[0017] These computer program instructions may be stored in a computer-readable medium that can guide a computer, other programmable data processing device, or other device that functions in a particular manner, thereby causing the instructions stored in the computer-readable medium to produce a manufactured article containing instructions that perform functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0018] Computer program instructions can also be loaded into a computer, other programmable data processing device, or other device to execute a series of operational steps on that computer, other programmable data processing device, or other device, thereby producing a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for performing a function / action specified in one or more blocks of a flowchart and / or block diagram.

[0019] Comprehensive solution Figure 1 is a schematic diagram showing an example of a voice communication system to which the embodiments of this specification can be applied.

[0020] As shown in Figure 1, User A operates communication terminal A, and User B operates communication terminal B. In a voice communication session, Users A and B converse with each other through their respective communication terminals A and B. Communication terminals A and B are connected via a data link 10. The data link 10 may be implemented as a point-to-point connection or a communication network. On either User A or User B's side, packet loss detection (not shown) is performed on voice packets transmitted from the other side. If packet loss is detected, packet loss compensation (PLC) can be performed to compensate for the packet loss, thereby making the reconstructed voice signal sound more complete and with fewer artifacts caused by packet loss.

[0021] Figure 2 is a schematic diagram of another example of a voice communication system to which the embodiments of this specification can be applied. In this example, users can conduct voice conferences with each other. As shown in Figure 2, User A operates communication terminal A, User B operates communication terminal B, and User C operates communication terminal C. In an audio conferencing session, Users A, B, and C converse with each other through their respective communication terminals A, B, and C. The communication terminals shown in Figure 2 have the same functionality as those shown in Figure 1, except that communication terminals A, B, and C are connected to the server via a common data link 20 or separate data links 20. The data links 20 may be implemented as point-to-point connections or communication networks. On any side of User A, User B, or User C, packet loss detection (not shown) is performed on voice packets transmitted from the other one or two sides. If packet loss is detected, packet loss compensation (PLC) can be performed to compensate for the packet loss, thereby making the reconstructed voice signal sound more complete and with fewer artifacts caused by the packet loss.

[0022] Packet loss can occur anywhere along the path from the source communication terminal to the server, and anywhere along the path from the source communication terminal to the destination communication terminal. Therefore, packet loss detection (not shown) and PLC can also be performed on the server instead, or in addition to the above. To perform packet loss detection and PLC on the server, packets received by the server may be depacketized (not shown). Then, after PLC, the voice signal, compensated for packet loss, may be repacketized (not shown) and transmitted to the destination communication terminal. If there are two users conversing simultaneously (this can be determined using Voice Activity Detection (VAD) technology), the mixer 800 must perform a mixing operation to combine the two streams of speech signals into one before transmitting the speech signals of the two users to the destination communication terminal. This may be done after PLC, but before the packetization operation.

[0023] Although three communication terminals are shown in Figure 2, the system may have a moderately larger number of communication terminals connected to it. This specification aims to solve the packet loss problem of sound field signals by applying different compensation methods to the monaural and spatial components obtained by appropriate conversion techniques applied to sound field signals. Specifically, this specification relates to constructing an artificial signal during spatial audio transmission when packet loss occurs.

[0024] As shown in Figure 3, in one embodiment, a packet loss compensation (PLC) device is provided to compensate for packet loss within a stream of voice packets, and each voice packet contains at least one voice frame in a transmission format that includes at least one monaural component and at least one spatial component. The PLC device may include a first compensation unit 400 for creating at least one monaural component for the lost frame in the lost packet, and a second compensation unit 600 for creating at least one spatial component for that lost frame. The created at least one monaural component and the created at least one spatial component become a created frame and replace the lost frame.

[0025] As known from prior art, in order to enable transmission, the audio stream is converted and stored in a frame structure which may be called a "transmission format," then packetized into audio packets at the source communication terminal, and subsequently received by the receiver 100 at the server or destination communication terminal. To perform PLC, a first de-packetizing unit 200 is provided to depacket each audio packet into at least one frame containing at least one monaural component and at least one spatial component, and a packet loss detector 300 is provided to detect packet loss in the stream. The packet loss detector 300 may or may not be considered part of the PLC device. In the case of the source communication terminal, any technique may be employed to convert the audio stream into any suitable transmission format.

[0026] One example of a transmission format can be obtained using an adaptive transform, such as an adaptive orthogonal transform, which yields multiple monaural and spatial components. For example, an audio frame may be a parameter-specific signal encoded based on parameter-specific decomposition, where at least one monaural component includes at least one eigenchannel component (such as at least one principal eigenchannel component), and at least one spatial component includes at least one spatial parameter. To give another example, an audio frame may be decomposed by principal component analysis (PCA), where at least one monaural component may include a signal based on at least one principal component, and at least one spatial component includes at least one spatial parameter.

[0027] Therefore, the source communication terminal may be equipped with a converter for converting the input audio signal into a parameter-specific signal. Depending on the format of the input audio signal, which may be called the "input format," the converter may be implemented using various technologies.

[0028] For example, the input audio signal may be an ambisonic B-form signal, and the corresponding converter can perform an adaptive transformation, such as the KLT (Karhunen-Loeve transform), on the B-form signal to obtain a parameter-specific signal consisting of intrinsic channel components (which may also be called rotated audio signals) and spatial parameters. Typically, an LRS (Left, Right, and Surround) signal or other artificially upmixed signal can be converted to a first-order ambisonic format (B-form), i.e., a WXY sound field signal (which may also be a WXYZ sound field signal, but in audio communications with LRS acquisition, only horizontal WXY is considered), and the adaptive transformation can combine all three channels W, X, and Y of the sound field signal and encode them into a new set of intrinsic channel components (rotated audio signals) Em (m=1, 2, 3) (i.e., E1, E2, E3, where the number of m may be greater or less) in order of increasing importance of information. The transformation can typically be described by a 3x3 transformation matrix (such as a covariance matrix) when there are three intrinsic signals, using a set of three spatial side parameters (d, φ, and θ) sent as side information, thus allowing the decoder to apply the inverse transformation to reconstruct the original sound field signal. Note that if packet loss occurs during transmission, neither the intrinsic channel components (rotated audio signals) nor the spatial side parameters can be obtained by the decoder.

[0029] Alternatively, the LRS signal may be directly converted to a parameter-specific signal. The coding structure described above may be called adaptive transform coding. As previously stated, coding may be performed using any adaptive transform such as KLT, or any other framework such as direct transformation from LRS signal to parameter-specific signal, but this specification provides an example of a specific algorithm to transform an input audio signal into a parameter-specific signal. For details, please refer to the section "Forward and Inverse Adaptive Transformations of Audio Signals" in this specification.

[0030] In the adaptive transformation coding discussed above, if sufficient bandwidth is available, E1, E2, and E3 are all encoded within the frame and then packetized within the packet stream; this is called discrete coding. Conversely, if bandwidth is limited, other methods may be considered, but E1 is a perceptually meaningful / optimized monaural representation of the original sound field, while E2 and E3 can be reconstructed by computing pseudo-uncorrelated signals. In practical embodiments, a weighted combination of E1 and an uncorrelated version of E1 is preferred, in which case the uncorrelated version may simply be a delayed copy of E1, and the weighting coefficients may be calculated based on the ratio of bandwidth energy between E1 and E2, and between E1 and E3. This method may be called predictive coding. For further details, see the section "Forward and Inverse Adaptive Transformations of Audio Signals" in this specification.

[0031] Next, in the input audio stream, each frame contains a set of frequency domain coefficients (for E1, E2, and E3) for the monaural components, and quantized side parameters, which may be called spatial components or spatial parameters. The side parameters may include predictive parameters if predictive coding is applied. When packet loss occurs, in independent coding, both Em (m=1, 2, 3) and spatial parameters are lost in transmission, but in predictive coding, a lost packet results in the loss of predictive parameters, spatial parameters, and E1.

[0032] The operation of the first depacketization unit 200 is the reverse operation of the packetization unit at the source communication terminal, and a detailed explanation of this is omitted here. The packet loss detector 300 may use any existing technique to detect packet loss. A common method is to detect the sequential number obtained by depackaging packets / frames from the packets received by the first depacketization unit 200, where discontinuities in the sequential number indicate that the packet / frame corresponding to the dropped sequential number has been lost. The sequential number is usually an essential field in VoIP packet formats such as Real-time Transport Protocol (RTP) format. Currently, one packet generally contains one frame (generally 20ms), but one packet can contain two or more frames, or one frame can span multiple packets. If one packet is lost, all frames within that packet are lost. If one frame is lost, it is likely that one or more packets have been lost, and packet loss compensation is generally performed on a frame-by-frame basis. In other words, the PLC is for recovering (one or more) frames that were lost due to a lost packet. Therefore, in the context of this specification, packet loss is generally synonymous with frame loss, and solutions are generally described in terms of frames unless it is necessary to refer to packets, for example, to emphasize the number of frames lost within a lost packet. Furthermore, in the claims, the phrase "each voice packet containing at least one voice frame" should be interpreted to include situations in which one frame spans two or more packets, and correspondingly, the phrase "a lost frame in a lost packet" should be interpreted to include "at least partially lost frame spanning more than one packet" caused by at least one lost packet.

[0033] This specification proposes performing separate packet loss compensation operations for monaural and spatial components, and for this purpose, provides a first compensation unit 400 and a second compensation unit 600, respectively. The first compensation unit 400 may be configured to create at least one monaural component for a lost frame by duplicating the corresponding monaural component within an adjacent frame.

[0034] In the context of this specification, “adjacent frame” means a frame that is immediately before or after the current frame (which may be a lost frame), or a frame that is separated by one or more other frames. In other words, either a future frame or a past frame can be used to reconstruct a lost frame, and generally the most recent future or past frame can be used. The most recent past frame may be called “the last frame.” In a modified example, an attenuation coefficient can be used when replicating the corresponding mono component.

[0035] If there are at least two consecutive frames that have been lost, the first compensation unit 400 may be configured to duplicate (one or more) past frames or (one or more) future frames for the earlier or later of the two consecutive frames that have been lost. In other words, the first compensation unit can create at least one monaural component for at least one earlier lost frame by duplicating the corresponding monaural component in an adjacent past frame, with or without using an attenuation coefficient, and can create at least one monaural component for at least one later lost frame by duplicating the corresponding monaural component in an adjacent future frame, with or without using an attenuation coefficient.

[0036] In the case of the second compensation unit 600, it may be configured to create at least one spatial component for the loss frame by smoothing the value of at least one spatial component in (one or more) adjacent frames, or by duplicating the corresponding spatial component in the last frame. As a variation, the first compensation unit 400 and the second compensation unit may employ different compensation methods.

[0037] In some contexts where delay may or may be acceptable, future frames may be used to help calculate the spatial components of the loss frame. For example, an interpolation algorithm may be used. That is, the second compensation unit 600 may be configured to create at least one spatial component for the loss frame via an interpolation algorithm based on the values ​​of the corresponding spatial components in at least one adjacent past frame and at least one adjacent future frame.

[0038] If at least two packets or at least two frames are lost, the spatial components of the total lost frames may be determined based on an interpolation algorithm. As mentioned above, there are various possible input and transmission formats. Figure 4 shows an example using parameter-specific signals as the transmission format. As shown in Figure 4, the audio signal is encoded and transmitted as a parameter-specific signal containing intrinsic channel components as monaural components and spatial parameters as spatial components (for details on the encoding side, see the section on "Forward and Inverse Adaptive Transformations of Audio Signals"). Specifically, the example includes three intrinsic channel components Em (m=1, 2, 3) and their corresponding spatial parameters, such as diffusion d (direction of E1), azimuth φ (horizontal direction of E1), and θ (rotation of E2 and E3 around E1 in 3D space). In the case of a successfully transmitted packet, both the intrinsic channel components and spatial parameters are transmitted normally (within the packet), whereas in the case of a lost packet / frame, both the intrinsic channel components and spatial parameters are lost, and the PLC is executed to create new intrinsic channel components and spatial parameters to replace those of the lost packet / frame. If the receiving communication terminal can directly reproduce (e.g., as binaural sound) the intrinsic channel components and spatial parameters that are successfully transmitted or created, or can first convert them to an appropriate intermediate output format, this intermediate output format may undergo further conversions or be reproduced directly. Like the input format, the intermediate output format may be any viable format, such as ambisonic B format (WXY or WXYZ sound field signals), LRS, or other formats. The audio signal in the intermediate output format may be reproduced directly or undergo further conversions to adapt to the playback device. For example, parameter-specific signals may be converted to WXY sound field signals via inverse adaptive conversion such as an inverse KLT (see the section "Forward and Inverse Adaptive Conversions of Audio Signals" in this specification), and then further converted to binaural audio signals if binaural reproduction is required. Accordingly, the packet loss compensation device of this specification may include a second inverse converter to perform inverse adaptive conversion on audio packets (receiving possible PLCs) to obtain inversely converted sound field signals.

[0039] In Figure 4, the first compensation unit 400 (Figure 3) can use conventional monaural PLCs, such as duplication with or without an attenuation coefficient, as described above and shown below.

[0040]

number

[0041]

number

[0042]

number

[0043] In addition to channel compensation, spatial compensation is also important. In the example shown in Figure 4, the spatial parameters may consist of d, φ, and θ. The stability of the spatial parameters is extremely important in maintaining perceptual continuity. Therefore, the second compensation unit 600 (Figure 3) may be configured to directly smooth the spatial parameters. Smoothing can be performed using any smoothing method, for example, by calculating the average value of past values.

[0044]

number

[0045] The stability of the spatial parameters is relatively high, for example, the d of the current frame p. p In some contexts where a large value is detected, simple replication of spatial parameters can be effective, but in the context of PLC, it is an even more effective method.

[0046]

number

[0047] In the embodiments and examples described above, since the intrinsic channel components do not contain any spatial information, the risk of spatial distortion caused by inappropriate compensation is reduced. Power converter for mono components Figure 4 depicts an example of a region PLC encoded within an independently encoded bitstream, in which case all intrinsic channel components E1, E2, and E3, and all spatial parameters, namely d, φ, and θ, need to be transmitted and, if necessary, restored for the PLC.

[0048] Compensation for independently coded regions is considered only if there is sufficient bandwidth for coding E1, E2, and E3. Otherwise, the frame may be coded using a predictive coding framework. In predictive coding, only one intrinsic channel component, i.e., the primary intrinsic channel E1, is actually transmitted. On the decoding side, other intrinsic channel components such as E2 and E3 are predicted using prediction parameters, for example, a2 and b2 for E2, and a3 and b3 for E3 (see the section "Forward and Inverse Adaptive Transforms of Speech Signals" in this specification for details on predictive coding). As shown in Figure 6, in this background, separate types of uncorrelated transformers are provided for E2 and E3 (transmitted or restored for PLC). Therefore, as long as E1 is successfully transmitted or restored (in PLC), the other two channels E2 and E3 can be directly predicted / constructed via a combination of uncorrelated transformers. This predictive PLC process eliminates nearly two-thirds of the computational load by adding only one calculation of the prediction parameters. Furthermore, since E2 and E3 do not need to be transmitted, bitrate efficiency is improved. The rest of Figure 6 is the same as that of Figure 4.

[0049] Therefore, in a modified embodiment of the packet loss compensation device, which is a feature of the first compensation unit 400 as shown in Figure 5, if each audio frame further includes at least one prediction parameter used for prediction based on at least one monaural component in the frame and at least one other monaural component in the frame, the first compensation unit 400 may have two sub-compensation units for performing PLC on the monaural component and the prediction parameter, respectively, namely a main compensation unit 408 for creating at least one monaural component for the loss frame and a third compensation unit 414 for creating at least one prediction parameter for the loss frame.

[0050] The main compensation unit 408 can function in the same way as the first compensation unit 400 discussed above. In other words, the main compensation unit 408 can be considered the core part of the first compensation unit 400 for creating some monaural component in the loss frame, and here it is configured solely for creating the main monaural component.

[0051] The third compensation unit 414 can operate in the same manner as the first compensation unit 400 or the second compensation unit 600. That is, the third compensation unit is configured to create at least one predictive parameter for a lost frame by duplicating the corresponding predictive parameter in the last frame, with or without using an attenuation coefficient, or by smoothing the values ​​of the corresponding predictive parameter in (one or more) adjacent frames. Assuming that frames i+1, i+2, ..., j-1 are lost, the lost predictive parameter in frame k can be smoothed as follows:

[0052]

number

[0053] In the case of a server with only one audio stream, mixing is unnecessary, and therefore predictive decoding does not necessarily need to be performed on the server. As a result, the created monaural component and the created predictive parameters can be directly packetized and forwarded to the destination communication terminal. In this case, predictive decoding is performed after depacketization, but before, for example, the inverse KLT shown in Figure 6.

[0054] In the case of a destination communication terminal, or when mixing operations for multiple audio streams are required within the server, the predictive decoder 410 (Figure 5) can predict other monaural components based on (one or more) monaural components created by the main compensation unit 408 and predictive parameters created by the third compensation unit 414. In fact, the predictive decoder 410 can also act on (one or more) successfully transmitted monaural components and (one or more) predictive parameters for successfully transmitted (unlossed) frames.

[0055] In general, the predictive decoder 410 can predict another monaural component using predictive parameters, based on the primary monaural component and its uncorrelated version within the same frame. Specifically, in the case of a loss frame, the predictive decoder can predict at least one other monaural component for the loss frame, based on one monaural component and its uncorrelated version, using at least one predictive parameter created. This operation can be expressed as follows:

[0056]

number

[0057] [Table 1]

[0058]

number

[0059]

number

[0060] Note that the operation of the predictive decoder 410 is the reverse process of the predictive coding of E2 and E3. For further details regarding the operation of the predictive decoder 410, see, but are not limited to, the section "Forward and Inverse Adaptive Transformations of Audio Signals" in this specification.

[0061] As mentioned above in equation (1), in the case of a lossy frame, the main mono component may be created simply by duplicating the main mono component in the last frame, that is,

[0062]

number

[0063] The solution combining equations (1') and (5') may be somewhat effective, but it has several drawbacks. From equations (1') and (5'), we can derive the following:

[0064]

number

[0065]

number

[0066]

number

[0067]

number

[0068] [Table 2] To avoid this recorrelation, it is necessary to avoid repetition or duplication. To this end, a time-domain PLC is provided as shown in the embodiment in Figure 7 and the example in Figure 8.

[0069] As shown in Figure 7, the first compensation unit 400 may include a first converter 402 for converting at least one monaural component in at least one past frame prior to the lost frame into a time-domain signal, a time-domain compensation unit 404 for compensating for packet loss with respect to the time-domain signal to obtain a time-domain signal with packet loss compensation, and a first inverse converter 406 for converting the time-domain signal with packet loss compensation into the form of at least one monaural component to obtain a created monaural component corresponding to at least one monaural component in the lost frame.

[0070] The time-domain compensation unit 404 can be implemented using many existing techniques, such as simply replicating time-domain signals in past or future frames, which will be omitted here.

[0071] [Table 3] In the example above, compensating for the loss frame requires two previous frames because the encoding framework is multi-duplicate transform (MDCT). When using non-multi-duplicate transforms, time-domain frames and frequency-domain frames have a one-to-one correspondence. Therefore, compensating for the loss frame only requires one previous frame.

[0072] For E2 and E3, similar PLC operations may be performed, but this specification also provides several other solutions, which will be discussed in the following sections. The computational load of the PLC algorithm discussed above is relatively high. Therefore, in some cases, measures may be taken to reduce the computational load. One is to predict E2 and E3 based on E1, as will be discussed later, and another is to combine time-domain PLC with other simpler methods.

[0073] For example, if multiple consecutive frames are lost, some of the lost frames, generally the earlier lost frames, can be compensated using a time-domain PLC, while the remaining lost frames can be compensated using simpler methods, such as replicating the frequency domain of the transmission format. Therefore, the first compensation unit 400 may be configured to create at least one monaural component for at least one subsequent lost frame by replicating the corresponding monaural component in an adjacent future frame, with or without using an attenuation coefficient.

[0074] The above description considered both predictive coding / decoding of less important intrinsic channel components and time-domain PLCs that can be used for any one intrinsic channel component. Time-domain PLCs are proposed to avoid recorrelation in duplicate PLCs for audio signals employing predictive coding (such as predictive KLT coding), but they may be applied in other contexts. For example, time-domain PLCs may be used even for audio signals employing non-predictive (independent) coding.

[0075] Predictive PLC for mono components In one embodiment shown in Figures 9A, 9B, and 10, independent coding is employed, so each audio frame contains at least two monaural components such as E1, E2, and E3 (Figure 10). Similar to Figure 4, in the case of a lost frame, all intrinsic channel components are lost due to packet loss and require PLC processing. As shown in the example in Figure 10, major monaural components such as the major intrinsic channel component E1 can be created / restored within the usual compensation framework such as duplication or other frameworks such as the time-domain PLC discussed above, while other monaural components such as the less important intrinsic channel components E2 and E3 can be created / restored based on the major monaural components (as indicated by the dashed arrows in Figure 10) using a method similar to the predictive decoding discussed above, and this method may therefore be called "predictive PLC". The rest of Figure 10 is the same as that of Figure 4, so a detailed explanation of it is omitted here.

[0076] Specifically, by using the following variations of equations (5), (5'), and (5''), it is possible to predict less important monaural components with or without adding the attenuation coefficient g.

[0077]

number

[0078]

number

[0079]

number

[0080] Accordingly, according to the embodiment, the first compensation unit 400 may include, as shown in Figure 9, a main compensation unit 408 for creating one of at least two monaural components for a loss frame, a predictive parameter calculator 412 for calculating at least one predictive parameter for a loss frame using past frames, and a predictive decoder 410 for predicting the other at least one of the at least two monaural components of the loss frame based on one monaural component created using the created at least one predictive parameter.

[0081] The main compensation unit 408 and the predictive decoder 410 are the same as those in Figure 5, and a detailed explanation of them is omitted here. The prediction parameter calculator 412 may be implemented by any technique, but in one modified embodiment, we propose calculating the prediction parameters using the last frame before the lost frame. The following equations illustrate a specific example, but are not limiting to this specification.

[0082]

number

[0083]

number

[0084] In other words, the predictive parameter calculator 412 may be implemented in the same way as the parameter coding unit 104, and this will be described later. To avoid abrupt fluctuations in the estimated parameters, the predicted parameters estimated above may be smoothed using some technique. A specific example is "ducker" style energy adjustment, which is represented by duck() in the following equation, thus avoiding rapid changes in the compensated signal level, especially in transition areas between speech and silence, or between speech and music.

[0085]

number

[0086]

number

[0087] A brief explanation of the calculation of prediction parameters has been given above, but this specification is not limited thereto. In fact, many more variations can be considered by referring to algorithms such as those discussed in the section on "Forward and Inverse Adaptive Transformations of Speech Signals."

[0088] In one modification, as shown in Figure 9A, a third compensation unit 414 may be further provided, which is similar to the third compensation unit discussed in the previous section and used to compensate for the lost prediction parameters in the predictive coding framework. Therefore, if at least one prediction parameter has been calculated for the last frame prior to the loss frame, the third compensation unit 414 can create at least one prediction parameter for the loss frame based on at least one prediction parameter for the last frame. Note that the solution shown in Figure 9A can also be applied to the predictive coding framework. In other words, the solution in Figure 9A is generally applicable to both predictive coding and non-predictive coding frameworks. In the predictive coding framework (where one or more predictive parameters exist in past frames that were successfully transmitted), the third compensation unit 414 operates in a non-predictive coding framework for the first lost frame (without adjacent past frames containing predictive parameters), and the predictive parameter calculator 412 operates in a non-predictive coding framework for one or more lost frames following the first lost frame, but either the predictive parameter calculator 412 or the third compensation unit 414 can operate.

[0089] Therefore, in Figure 9A, the prediction parameter calculator 412 may be configured to calculate at least one prediction parameter for the loss frame using previous frames if no prediction parameters are included or have not been created / calculated for the last frame prior to the loss frame, and the prediction decoder 410 may be configured to use the calculated or created at least one prediction parameter to predict the other mono component of at least one of at least two mono components for the loss frame based on one created mono component.

[0090] As discussed above, the third compensation unit 414 may be configured to create at least one predictive parameter for the loss frame by duplicating the corresponding predictive parameter in the last frame, or by smoothing the values ​​of the corresponding predictive parameter in (one or more) adjacent frames, or by interpolation using the values ​​of the corresponding predictive parameter in past and future frames, with or without using an attenuation coefficient.

[0091] In further variations, such as those shown in Figure 9B, the predictive PLC discussed in this section can be combined with non-predictive PLCs (including the simple replication or PLC framework discussed with reference to Figure 7, as well as those discussed in the "Comprehensive Solutions" section). In other words, both non-predictive and predictive PLCs can be run on low-importance monaural components, and the results obtained can be combined to obtain the final monaural component, such as a weighted average of the two results. This process can also be considered as adjusting one result with the other, and the weighting coefficients can be set according to the specific background and which is dominant.

[0092] Therefore, as shown in Figure 9B, in the first compensation unit 400, the main compensation unit 408 may be further configured to create at least one other mono component, and the first compensation unit 400 further includes an adjustment unit 416 for adjusting at least one other mono component predicted by the predictive decoder 410 with at least one other mono component created by the main compensation unit 408.

[0093] PLC for spatial components In the "Comprehensive Solution" section, we considered PLC for spatial components such as spatial parameters d, φ, and θ. The stability of spatial parameters is extremely important in maintaining perceptual continuity. This is achieved by directly smoothing the parameters in the "Comprehensive Solution" section. As another independent solution, or as a way to supplement the PLC considered in the "Comprehensive Solution" section, the smoothing operation on spatial parameters can be performed on the encoding side. In this way, since the spatial parameters are smoothed on the encoding side, the PLC results for the spatial parameters on the decoding side become even smoother and more stable.

[0094] Similarly, the smoothing operation may be performed directly on the spatial parameters. However, this specification further proposes smoothing the spatial parameters by smoothing the elements of the transformation matrix derived from the spatial parameters.

[0095] As discussed in the "Comprehensive Solution" section, the monaural and spatial components can be derived using adaptive transformations, one important example being KLT, which has already been discussed. In such transformations, the input format (such as WXY or LRS) may be transformed into a rotated audio signal (such as the intrinsic channel components when encoding with KLT) via a transformation matrix, such as the covariance matrix used when encoding with KLT. Furthermore, the spatial parameters d, φ, and θ are derived from the transformation matrix. Therefore, if the transformation matrix is ​​smoothed, the spatial parameters are also smoothed.

[0096] Here, various smoothing operations such as moving averages or historical averages, as shown below, can be applied.

[0097]

number

[0098] Therefore, as shown in Figure 11, a second converter 1000 is provided for converting the input spatial audio signal into a transmission frame. Here, each frame comprises at least one monaural component and at least one spatial component. The second converter may include an adaptive converter 1002 for decomposing each frame of the input spatial audio signal into at least one monaural component associated with the frame of the input spatial audio signal via a transformation matrix, a smoothing unit 1004 for smoothing the values ​​of each element of the transformation matrix to obtain a smoothed transformation matrix for the current frame, and a spatial component extractor 1006 for deriving at least one spatial component from the smoothed transformation matrix.

[0099] Smoothing the covariance matrix significantly improves the stability of the spatial parameters. This allows for simpler replication of the spatial parameters, which is an effective and more efficient method in the context of PLC, as discussed in the "Comprehensive Solution" section.

[0100] Further details on smoothing the covariance matrix and deriving spatial parameters from it are described in the section on "Forward and Inverse Adaptive Transformations of Audio Signals." Forward and reverse adaptive conversion of audio signals This section provides some examples of how to obtain an audio frame and corresponding audio encoders and decoders in a transmission format such as a parameter-specific signal, which serves as an example audio signal to address the purposes of this specification. However, this specification is not explicitly limited thereto. The PLC apparatus and methods discussed above may be located or implemented in a server or the like before the audio decoder, or they may be incorporated into an audio decoder located in a destination communication terminal or the like.

[0101] To further clarify this section, some terms will not be exactly the same as those used in the previous section, but their correspondences will be addressed below as needed. A two-dimensional spatial sound field is typically captured by a three-microphone array ("LRS") and then represented in a two-dimensional B-form ("WXY"). The two-dimensional B-form ("WXY") is an example of a sound field signal, and in particular an example of a three-channel sound field signal. The two-dimensional B-form typically represents the sound field in the X and Y directions, but not the sound field in the Z direction (height). Such a three-channel spatial sound field signal can be encoded using an independent parameter method. The independent method has been shown to be effective at relatively high operating bit rates, while the parameter method has been shown to be effective at relatively low rates (e.g., less than 24 kbit / s per channel). This section describes an encoding system using the parameter method.

[0102] Parameter coding offers new advantages in the layered transmission of sound field signals. Parameter coding techniques typically involve generating a down-mix signal and spatial parameters describing one or more spatial signals. Parameterized description of spatial signals generally requires a lower bitrate than the bitrate required in the background of independent coding. Therefore, given the constraints on a given bitrate, parameter coding allows for more bits to be used for independent coding of the down-mix signal, and the sound field signal can be reconstructed from the down-mix signal using a set of spatial parameters. Thus, the down-mix signal can be coded at a higher bitrate than the bitrate used to code each channel of the sound field signal separately. As a result, the down-mix signal may have higher perceptual quality. This feature of parameter coding of spatial signals is beneficial in applications involving layered coding, particularly in teleconferencing systems where monaural and spatial clients (or terminals) coexist. For example, in the case of a monaural client, the down-mix signal can be used to render a monaural output (ignoring the spatial parameters used to reconstruct the complete sound field signal). In other words, the bitstream for a monaural client can be obtained by removing bits from the complete sound field bitstream related to spatial parameters.

[0103] The idea behind the parameter-based approach is to send a monaural downmix signal with a set of spatial parameters that allow the decoder to reconstruct a perceptually appropriate approximation of the (3-channel) sound field signal. The downmix signal can be derived from the sound field signal to be encoded using non-adaptive downmixing and / or adaptive downmixing techniques.

[0104] Non-adaptive methods for deriving a downmix signal may include the use of a fixed reversible transform. One example of such a transform is a matrix that converts the "LRS" notation to a two-dimensional B-form ("WXY"). In this case, component W may be a reasonable choice for the downmix signal due to the physical properties of component W. The "LRS" representation of the sound field signal can be assumed to be captured by an array of three microphones, each array having a cardioid polar pattern. In such a case, the W component of the B-form representation corresponds to the signal captured by a (hypothetical) omnidirectional microphone. The hypothetical omnidirectional microphone provides a signal that is substantially unresponsive to the spatial position of the sound source, thus providing a robust and stable downmix signal. For example, the angular position of the main sound source represented by the sound field signal does not affect the W component. The conversion to B-form is reversible, and the "LRS" representation of the sound field can be reconstructed with "W" and the other two components, namely "X" and "Y". Therefore, (parameter-based) encoding may be performed in the "WXY" region. More generally, the aforementioned "LRS" region may be called the captured region, that is, the region in which the sound field signal is captured (using a microphone array).

[0105] The advantage of parameter coding using non-adaptive downmixing lies in the fact that, because downmixed signals are stable and robust, such non-adaptive methods provide a robust foundation for prediction algorithms implemented in the "WXY" domain. A potential drawback of parameter coding using non-adaptive downmixing is that it typically involves a lot of noise and reverberation. Therefore, prediction algorithms implemented in the "WXY" domain may perform poorly because the "W" signal typically has different characteristics from the "X" and "Y" signals.

[0106] Adaptive techniques for creating downmix signals may include performing an adaptive transformation of the "LRS" representation of the sound field signal. One example of such a transformation is the Karhunen-Loeve transform (KLT). This transformation is derived by performing eigenvalue decomposition of the inter-channel covariance matrix of the sound field signal. In the case considered, the inter-channel covariance matrix in the "LRS" region may be used. Next, the "LRS" representation of the signal can be transformed into a set of eigenchannels using the adaptive transformation, which can be denoted as "E1 E2 E3". High coding gain can be achieved by applying coding to the "E1 E2 E3" representation. In the case of a parameter coding technique, the "E1" component can serve as the monaural downmix signal.

[0107] The advantage of this adaptive downmixing framework is that the intrinsic region is favorable for encoding. In principle, an optimal trade-off between rate and distortion can be achieved when encoding the intrinsic channel (or intrinsic signal). In an ideal case, the intrinsic channels are completely uncorrelated and can be encoded independently of each other without performance loss (compared to combined encoding). Moreover, signal E1 is usually less noisy and typically contains less reverberation than the "W" signal. However, adaptive downmixing also has drawbacks. The first drawback relates to the fact that the adaptive downmixing transformation must be recognized by the encoder and decoder, and therefore, parameters that are indicators of the adaptive downmixing transformation must be encoded and transmitted. To achieve the goal of uncorrelatedizing the intrinsic signals E1, E2, and E3, the adaptive transformation needs to be updated relatively frequently. Periodically updating the adaptive transmission increases computational complexity and requires a bitrate to transmit the transformation description to the decoder.

[0108] A second drawback of parameter coding based on adaptive methods can stem from the instability of the E1 downmix signal. This instability may be due to the fact that the underlying transformation providing the downmix signal E1 is signal-adaptive, and therefore the transformation changes over time. Variations of KLT typically depend on the spatial characteristics of the signal source. Thus, depending on the type of input signal, it can be particularly difficult in backgrounds with multiple speakers where speakers are complexly represented in the sound field signal. Another cause of instability in adaptive methods can stem from the spatial characteristics of the microphone used to capture the "LRS" representation of the sound field signal. Typically, a directional microphone array with a polar pattern (e.g., cardioid) is used to capture the sound field signal. In such cases, the inter-channel covariance matrix of the sound field signal represented as "LRS" can change significantly if the spatial characteristics of the signal source change (e.g., in a background with multiple speakers), and the results from KLT are similar.

[0109] This specification describes a downmixing technique that addresses the stability issues of the adaptive downmixing technique described above. The downmixing framework described combines the advantages of both adaptive and non-adaptive downmixing methods. In particular, it proposes revealing an adaptive downmix signal, such as a "beamformed" signal, which mainly contains the dominant component of the sound field signal and maintains the stability of the downmix signal derived using a non-adaptive downmixing method.

[0110] It should be noted that the conversion from "LRS" representation to "WXY" representation is reversible, but not orthonormal. Therefore, in the context of encoding (for example, due to quantization), the application of KLT in the "LRS" domain is not always the same as the application of KLT in the "WXY" domain. The advantage of WXY representation lies in the inclusion of a component "W" that is robust in terms of the spatial characteristics of the sound source. In "LRS" representation, all components usually respond equally to the spatial variability of the sound source. Conversely, the "W" component of WXY representation is usually independent of the angular position of the main sound source in the sound field signal.

[0111] Furthermore, regardless of the representation of the sound field signal, it is beneficial to apply KLT in the transformed region where at least one component of the sound field signal is spatially stable. Thus, it can be beneficial to transform the representation of the sound field into a region where at least one component of the sound field signal is spatially stable. Subsequently, an adaptive transform (such as KLT) may be used in the region where at least one component signal is spatially stable. In other words, the use of a non-adaptive transform, which depends only on the characteristics of the polar pattern of the microphones in the microphone array used to capture the sound field array, is combined with an adaptive transform, and this transform depends on the time-varying covariance matrix between channels of the sound field signal in the non-adaptive transform region. Note that both transforms (i.e., non-adaptive and adaptive transforms) are reversible. In other words, the benefit of combining the two proposed transforms is that it is guaranteed that both transforms are reversible in all cases, and therefore these two transforms enable effective encoding of the sound field signal.

[0112] Thus, we propose converting the sound field signal acquired from the acquired region (e.g., the "LRS" region) into a non-adaptive conversion region (e.g., the "WXY" region). Subsequently, an adaptive conversion (e.g., KLT) can be calculated based on the sound field signal in the non-adaptive conversion region. The sound field signal may be converted into an adaptive conversion region (e.g., the "E1E2E3" region) using the adaptive conversion (e.g., KLT).

[0113] The following describes various frameworks for parameter coding. These coding frameworks can utilize predictive and / or KLT-based parameterization. The goal is to improve the overall trade-off between codec rate and quality by combining these parameter coding frameworks with the downmixing frameworks described above.

[0114] Figure 22 is a block diagram of an exemplary encoding system 1100. The illustrated system 1100 comprises components 120 typically found inside the encoder of the encoding system 1100 and components 130 typically found inside the decoder of the encoding system 1100. The encoding system 1100 comprises a (reversible and / or non-adaptive) conversion unit 101 from the "LRS" region to the "WXY" region, followed by an energy-concentrated orthonormal (adaptive) conversion unit 102 (e.g., KLT conversion). The sound field signal 110 in the region of the acquisition microphone array (e.g., the "LRS" region) is converted to a sound field signal 111 by the non-adaptive conversion unit 101 in a region containing a stable downmix signal (e.g., the signal "W" in the "WXY" region). Subsequently, the sound field signal 111 is converted to a sound field signal 112 containing uncorrelated channels or signals (e.g., channels E1, E2, E3) using the uncorrelated conversion unit 102.

[0115] The first intrinsic channel E1 113 can be used to parameterize the other intrinsic channels E2 and E3 (parametric coding, also referred to as “predictive coding” in the preceding section). However, this specification is not limited thereto. In another embodiment, E2 and E3 cannot be parameterized and are simply coded in the same way as E1 (an independent method, also referred to as “non-predictive / independent coding” in the preceding section). The downmix signal E1 may be coded using a downmixing coding unit 103 within the framework of single-channel voice and / or speech coding. The decoded downmix signal 114 (which is also available in the corresponding decoder) can be used to parameterize the intrinsic channels E2 and E3. Parameter coding may be performed in a parameter coding unit 104. The parameter coding unit 104 can provide a set of predictive parameters, which may be used to reconstruct signals E2 and E3 from the decoded signal E1 114. This reconstruction is typically performed in the corresponding decoder. Furthermore, the decoding operation includes using the reconstructed E1 signal and the parameter-decoded E2 and E3 signals (code 115), as well as performing an inverse orthonormal transform (e.g., inverse KLT) 105 to bring the reconstructed sound field signal 116 into a non-adaptive transform region (e.g., the "WXY" region). Following the inverse orthonormal transform 105, a transform 106 (e.g., an inverse non-adaptive transform) is performed to bring the reconstructed sound field signal 117 into the captured region (e.g., the "LRS" region). Transform 106 is typically the inverse transform of transform 101. The reconstructed sound field signal 117 may be rendered by a terminal of a video conferencing system configured to render sound field signals. A monaural terminal of a video conferencing system can directly render the reconstructed downmix signal E1114 (without needing to reconstruct the sound field signal 117).

[0116] To achieve high-quality encoding, it is beneficial to apply parameter coding in the subband domain. Time-domain signals can be converted to the subband domain using time-frequency (TF) transforms, such as duplicate TF transforms like the Modified Discrete Cosine Transform (MDCT). Since transforms 101 and 102 are linear, the TF transform can, in principle, be applied equally to the captured domain (e.g., the "LRS" domain), the non-adaptive transform domain (e.g., the "WXY" domain), or the adaptive transform domain (e.g., the "E1E2E3" domain). Thus, the encoder may include a unit configured to perform the TF transform (e.g., unit 201 in Figure 23A).

[0117] The frame description of the 3-channel sound field signal 110 generated using the encoding system 1100 includes, for example, two components. One component includes parameters that are applied at least on a frame-by-frame basis. The other component includes a description of the monaural waveform obtained based on the downmix signal 113 (e.g., E1) by using a 1-channel, monaural coder (e.g., a voice and / or speech coder based on transformation).

[0118] The decoding operation involves decoding a single-channel monaural downmix signal (e.g., the E1 downmix signal). The reconstructed downmix signal 114 is then used to reconstruct the remaining channels (e.g., the E2 and E3 signals) using parameterization parameters (e.g., prediction parameters). Subsequently, the reconstructed intrinsic signals E1, E2, and E3 115 are alternately returned to the non-adaptive transformation region (e.g., the "WXY" region) using transmitted parameters describing the decorrelation of the transformation 102 (e.g., the KLT parameters). The reconstructed sound field signal 117 within the captured region may be obtained by converting the "WXY" signal 116 back to the original "LRS" region 117.

[0119] Figures 23A and 23B are more detailed block diagrams of the exemplary encoder 1200 and exemplary decoder 250, respectively. In the illustrated example, the encoder 1200 includes a TF converter 201 configured to convert the sound field signal 111 (or its channels) in the non-adaptive conversion domain to the frequency domain, thereby yielding subband signals 211 to the sound field signal 111. Thus, in the illustrated example, the conversion 202 of the sound field signal 111 to the adaptive conversion domain is performed with the different subband signals 211 of the sound field signal 111.

[0120] The following describes the various components of the encoder 1200 and the decoder 250. As described above, the encoder 1200 may include a first conversion unit 101 configured to convert the sound field signal 110 obtained from the captured region (e.g., the "LRS" region) into a sound field signal 111 within a non-adaptive conversion region (e.g., the "WXY" region). The conversion from the "LRS" region to the "WXY" region is performed by the conversion [WXY] T =M(g)[LRS] T This can be done by the following, and the transformation matrix M(g) can be obtained by the following:

[0121]

number

[0122] The KLT102 provides rate distortion when it can adapt sufficiently frequently to the time-varying statistical characteristics of the signal to which it is applied. However, frequent adaptation of the KLT can lead to coding artifacts, which degrade perceptual quality. Experiments have shown that a good balance between rate distortion and the resulting artifacts can be achieved by applying the KLT transform to the sound field signal 111 in the "WXY" region, rather than applying the KLT transform to the sound field signal 110 in the "LRS" region (as already mentioned above).

[0123] The parameter g of the transformation matrix M(g) may be useful in the sense of stabilizing the KLT. As described above, it is desirable for the KLT to be substantially stable. By selecting g≠sqrt(2), the transformation matrix M(g) is not orthogonal, and the W component becomes prominent (when g>sqrt(2)) or less prominent (when g<sqrt(2)). This may have a stabilizing effect on the KLT. In any case where g≠0, the transformation matrix M(g) is always invertible, and thus encoding becomes easier (due to the existence of the inverse matrix M -1 (g) and it can be used in the decoder 250). However, if g≠sqrt(2), the encoding efficiency usually decreases (in terms of the rate - distortion trade - off) because the transformation matrix M(g) is not orthogonal. Therefore, the parameter g should be selected to improve the trade - off between the encoding efficiency and the stability of the KLT. In the course of experiments, it has been revealed that with g = 1 (thus a "proper" transformation to the "WXY" region), the trade - off between the encoding efficiency and the stability of the KLT becomes reasonable.

[0124] In the next step, the sound field signal 111 in the "WXY" region is analyzed. First, the inter - channel covariance matrix may be estimated using the covariance estimator 203. This estimation may be performed in the sub - band region (as shown in FIG. 23A). The covariance estimator 203 may include a smoothing process aimed at improving the estimation of the inter - channel covariance and reducing (e.g., minimizing) the problems that may occur because the estimation can vary substantially over time. Thus, the covariance estimator 203 may be configured to perform smoothing of the covariance matrix of the frames of the sound field signal 111 along the timeline.

[0125] Furthermore, the covariance estimation unit 203 may be configured to decompose the inter-channel covariance matrix using eigenvalue decomposition (EVD), which yields an orthonormal transformation V that diagonalizes the covariance matrix. The transformation V makes it easy to rotate the "WXY" channels to an eigenregion containing the eigenchannels "E1 E2 E3", which is given by the following equation.

[0126]

number

[0127]

number

[0128]

number

[0129]

number

[0130] In principle, conversion

[0131]

number

[0132] Parameter coding in the KLT domain may be performed as follows: Waveform coding can be applied to the intrinsic signal E1 (single monaural encoder 103). Furthermore, parameter coding may be applied to intrinsic signals E2 and E3. In particular, two uncorrelated signals can be generated from intrinsic signal E1 using an uncorrelated method (e.g., using a delayed version of intrinsic signal E1). The energy of the uncorrelated version of intrinsic signal E1 may be adjusted so that its energy matches the energy of the corresponding intrinsic signals E2 and E3, respectively. As a result of the energy adjustment, energy adjustment gains b2 (for intrinsic signal E2) and b3 (for intrinsic signal E3) can be obtained. These energy adjustment gains (which may be considered together with a2 as predictive parameters) may be calculated as described below. Energy adjustment gains b2 and b3 may be calculated in the parameter estimation unit 205.

[0133] [Table 4] For example, to describe the subbands of the sound field signal 112 within the "E1 E2 E3" region, three parameters are used to describe the KLT. These are d, φ, and θ, plus two additional gain adjustment parameters, b2 and b3. Therefore, the total number of parameters is five per subband. If there are many more channels describing the sound field signal, KLT coding requires a much larger number of transformation parameters to describe the KLT. For example, the minimum number of transformation parameters required to identify the KLT in four-dimensional space is six. In addition, three adjustment gain parameters are used to calculate the intrinsic signals E2, E3, and E4 from the intrinsic signal E1. Therefore, the total number of parameters is nine per subband. In the general case, given a sound field signal containing M channels, describing the KLT transformation parameters takes O(M) 2The parameters are determined, and O(M) parameters are required to describe the energy adjustments performed on the intrinsic signal. Therefore, calculating the set of 212 transformation parameters (to describe KLT) for each subband may require encoding a considerable number of parameters.

[0134] This specification describes an efficient parameter coding framework in which the number of parameters used to encode a sound field signal is always O(M) (in particular, as long as the number of subbands N is substantially greater than the number of channels M). In particular, this specification proposes calculating KLT transformation parameters 212 for multiple subbands (e.g., for all subbands, or for all subbands containing frequencies higher than those contained within the starting band). Such a KLT calculated based on and applied to multiple subbands may be called a broadband KLT. The broadband KLT provides only completely uncorrelated eigenvectors E1, E2, E3 for a combined signal corresponding to multiple subbands, and the broadband KLT is determined based on these. On the other hand, when the broadband KLT is applied to individual subbands, the eigenvectors of these individual subbands are usually not completely uncorrelated. In other words, the broadband KLT produces mutually uncorrelated eigensignals only when considering the full-band version of the eigensignal. However, it can be seen that a considerable amount of correlation (redundancy) remains present on a subband basis. This correlation (redundancy) between eigenvectors E1, E2, and E3 at the subband level can be efficiently utilized by a prediction framework. Therefore, a prediction framework may be applied to predict eigenvectors E2 and E3 based on the primary eigenvector E1. Thus, we propose applying predictive coding to the eigenchannel representation of the sound field signal obtained using broadband KLT performed on the sound field signal 111 in the "WXY" region.

[0135] A prediction-based coding framework (or simply "predictive coding") can provide a parameterization that splits the parameterized signals E2, E3 into a fully correlated (predicted) component and an uncorrelated (unpredicted) component derived from the downmix signal E1. The parameterization may be performed in the frequency domain after a suitable TF transform 201. Specific frequency bins of the transformed time frame of the sound field signal 111 can be combined to form a frequency band that is processed together as a single vector (i.e., a subband signal). Typically, this frequency band is the one that provides perceptual stimulation. The bandwidth of the frequency bins can be induced to be only one or two frequency bands across the entire frequency range of the sound field signal.

[0136] More specifically, in each time frame p (for example, 20ms) and for each frequency band k, the eigenvector E1(p,k) can be used as the downmix signal 113, and the eigenvectors E2(p,k) and E3(p,k) can be reconstructed as follows:

[0137]

number

[0138] [Table 5]

[0139]

number

[0140] The calculation of the uncorrelated components of the intrinsic signals E2 and E3 utilizes the calculation of two uncorrelated versions of the downmix signal E1 using the uncorrelators d2() and d3(). Typically, the quality (performance) of the uncorrelated signals d2(E1(p,k)) and d3(E1(p,k)) affects the overall perceptual quality of the proposed coding framework. Various uncorrelation methods may be used. For example, the frames of the downmix signal E1 may be an all-pass filtered to yield the corresponding frames of the uncorrelated signals d2(E1(p,k)) and d3(E1(p,k)).

[0141] [Table 6] If the uncorrelated signal is replaced with the remaining signal encoded in mono, the resulting system will again achieve waveform coding. This can be advantageous if the predictive gain is high. For example, one might consider explicitly calculating the remaining signals resE2(p,k)=E2(p,k)-a2(p,k)*E1(p,k)) and resE3(p,k)=E3(p,k)-a3(p,k)*E1(p,k)), which have the properties of an uncorrelated signal (at least in terms of the assumed model obtained by equations (17) and (18). Waveform coding of these signals resE2(p,k) and resE3(p,k) may be considered as an alternative to using a synthesized uncorrelated signal. Other instances of the mono codec may be used to perform explicit coding of the remaining signals resE2(p,k) and resE3(p,k), but this would be disadvantageous because the bitrate required to send the remaining signal to the decoder would be relatively high. On the other hand, the advantage of this method is that the allocated bitrate is larger, making it easier to reconstruct the decoder and approaching perfect reconstruction.

[0142] The energy adjustment gains b2(p,k) and b3(p,k) for an uncorrelated circuit can be calculated as follows.

[0143]

number

[0144] [Table 7] As described above, the uncorrelated signals d2() and d3() may be implemented as one-frame delays and two-frame delays, respectively. In this case, the aforementioned energy mismatch usually occurs (especially when the signal is transient). Further energy adjustments need to be performed (in encoder 1200 and / or decoder 250) to ensure the accuracy of the signal model obtained by equations (17) and (18), and to insert appropriate amounts of uncorrelated signals d2(E1(p,k)) and d3(E1(p,k)) during the reconstruction process.

[0145] In one example, further energy adjustments can be performed as follows: Encoder 1200 may insert energy adjustment gains b2(p,k) and b3(p,k) (calculated using equations (21) and (22)) into the spatial bitstream 221 (which may be in a quantized and encoded version).

[0146] [Table 8] In addition, the decoder 250 may be configured to generate an uncorrelated signal 264 (in the uncorrelated unit 252) based on the decoded downmix signal MD(p,k) 261, for example, using one or two frame delays (denoted as p-1 and p-2), which can be described as follows.

[0147]

number

[0148]

number

[0149]

number

[0150]

number

[0151]

number

[0152]

number

[0153]

number

[0154] The energy adjustment method described above requires only the energy of the downmix signal MD, which is decoded for each subband f (also called the parameter band k) for the current frame and the two previous frames, namely p, p-1, and p-2, as input.

[0155] It should be noted that the updated energy adjustment gains b2new(p,k) and b3new(p,k) may be calculated directly by the encoder 1200 and then decoded and inserted into the spatial bitstream 221 (in place of the energy adjustment gains b2(p,k) and b3(p,k)). This can be beneficial in terms of efficient encoding of the energy adjustment gains.

[0156] Thus, a frame of the sound field signal 110 may be described using a downmix signal E1 113, one or more sets of transformation parameters 213 that describe adaptive transformations (in this case, each set of transformation parameters 113 describes an adaptive transformation used for multiple subbands), one or more prediction parameters a2(p,k) and a3(p,k) for each subband, and one or more energy adjustment gains b2(p,k) and b3(p,k) for each subband. In addition to the prediction parameters a2(p,k) and a3(p,k) and the energy adjustment gains b2(p,k) and b3(p,k) (collectively referred to as prediction parameters as mentioned above), one or more sets of transformation parameters (these are the spatial parameters mentioned above) 213 may also be inserted into the spatial bitstream 221, and only this spatial bitstream may be decoded by a terminal of the video conferencing system, which is configured to render the sound field signal. Furthermore, the downmix signal E1 113 may be encoded using a (conversion-based) monaural audio and / or speech encoder 103. The encoded downmix signal E1 may be inserted into a downmixing bitstream 222, which may be decoded at a terminal of the video conferencing system, which is configured solely to render a monaural signal.

[0157] As noted above, this specification proposes calculating an uncorrelated transform 202 and applying it to multiple subbands. In particular, a broadband KLT (e.g., a single KLT per frame) can be used. Using a broadband KLT can be beneficial with respect to the perceptual characteristics of the downmix signal 113 (thus enabling the implementation of a layered video conferencing system). As stated above, the parameter coding may be based on predictions made in the subband region. This allows for fewer parameters to be used to describe the sound field signal than with parameter coding using a narrowband KLT, in which case a different KLT is calculated separately for each of the multiple subbands.

[0158] As described above, the prediction parameters may be quantized and encoded. Parameters directly related to the prediction may be conveniently encoded using frequency difference quantization followed by Huffman coding. Therefore, the parameterized description of the sound field signal 110 may be encoded using a variable bitrate. If an overall operating bitrate constraint is set, the rate required to parameterize a frame of a particular sound field signal can be subtracted from the total available bitrate, and the remaining 217 may be used for mono encoding of one channel of the downmix signal 113.

[0159] Figures 23A and 23B are block diagrams of an exemplary encoder 1200 and an exemplary decoder 250. The illustrated audio encoder 1200 is configured to encode a frame of a sound field signal 110 containing multiple audio signals (or audio channels). In the illustrated example, the sound field signal 110 has already been transformed from the captured region to a non-adaptive transformation region (i.e., the WXY region). The audio encoder 1200 includes a TF transformer 201 configured to transform the sound field signal 111 from the time domain to the subband domain, thereby yielding a subband signal 211 for various audio signals of the sound field signal 111.

[0160] The speech encoder 1200 includes transformation calculation units 203 and 204, which are configured to calculate an orthogonal transformation V (e.g., KLT) that compresses energy based on the frames of the sound field signal 111 in the non-adaptive transformation region (in particular, based on the subband signal 211). The transformation calculation units 203 and 204 may also include a covariance estimation unit 203 and a transformation parameter coding unit 204. Furthermore, the speech encoder 1200 includes a transformation unit 202 (also called an uncorrelated unit), which is configured to apply an orthogonal transformation V that compresses energy to the frames derived from the frames of the sound field signal (e.g., to the subband signal 211 of the sound field signal 111 in the non-adaptive transformation region). In this way, corresponding frames of the rotated sound field signal 112, including multiple rotated speech signals E1, E2, and E3, can be obtained. The rotated sound field signal 112 is sometimes referred to as the sound field signal 112 in the adaptive transformation region.

[0161] Furthermore, the audio encoder 1200 includes a waveform encoding unit 103 (also referred to as a monaural encoder or downmixing encoder), which is configured to encode the first rotated audio signal E1 (i.e., the main intrinsic signal E1) of the multiple rotated audio signals E1, E2, and E3. In addition, the audio encoder 1200 includes a parameter encoding unit 104 (also referred to as a parameter coding unit), which is configured to calculate a set of prediction parameters a2 and b2 and to calculate the second rotated audio signal E2 among the multiple rotated audio signals E1, E2, and E3 based on the first rotated audio signal E1. The parameter encoding unit 104 may be configured to calculate one or more other sets of prediction parameters a3 and b3 and to calculate one or more other rotated audio signals E3 among the multiple rotated audio signals E1, E2, and E3. The parameter coding unit 104 may include a parameter estimation unit 205 configured to estimate and encode a set of prediction parameters. Furthermore, the parameter coding unit 104 may include a prediction unit 206 configured to calculate the correlated and uncorrelated components of the second rotated audio signal E2 (and one or more other rotated audio signals E3) using, for example, the formulas described herein.

[0162] The audio decoder 250 in Figure 23B is configured to receive a spatial bitstream 221 (showing one or more sets of prediction parameters 215, 216 and one or more transformation parameters (spatial parameters) 212, 213, 214 describing the transformation V) and a downmixing bitstream 222 (showing the initially rotated audio signal E1 113 or its reconstructed version 261). The audio decoder 250 is configured to provide frames of the reconstructed sound field signal 117, which contains multiple reconstructed audio signals, from the spatial bitstream 221 and the downmixing bitstream 222.

[0163] [Table 9] Various modifications of the parameter coding framework described above may be implemented. For example, another operational form of the parameter coding framework allows for uncorrelated complete convolution without additional delay, by first generating two intermediate signals in the parameter domain by applying energy adjustment gains b2(p,k) and b3(p,k) to the downmix signal E1. Subsequently, an inverse TF transform can be performed on these two intermediate signals to yield two time-domain signals. Next, the two time-domain signals may be decorrelated. These decorrelated time-domain signals may be appropriately added to the reconstructed prediction signals E2 and E3. Thus, in the alternative implementation, the uncorrelated signals are generated in the time domain (not the subband domain).

[0164] As described above, the adaptive transformation 102 (e.g., KLT) may be calculated using the inter-channel covariance matrix of the frame for the sound field signal 111 in the non-adaptive transformation region. The advantage of applying KLT parameter coding on a subband basis is that the inter-channel covariance matrix can be accurately reconstructed by the decoder 250. However, this requires O(M) to determine the transformation V. 2 ) Encoding and / or transmission of conversion parameters will be required.

[0165] The parameter coding framework described above does not result in an accurate reconstruction of the inter-channel covariance matrix. Nevertheless, it has been observed that good perceptual quality can be achieved for two-dimensional sound field signals using the parameter coding framework described herein. However, it may be beneficial to reconstruct accurate coherence for all pairs of reconstructed eigensignals. This can be achieved by extending the parameter coding framework described above.

[0166] In particular, another parameter γ may be calculated and transmitted to describe the normal correlation between the intrinsic signals E2 and E3. This makes it possible for the decoder 250 to restore the original covariance matrix of the two prediction errors. As a result, the total covariance of the three-dimensional signal can be restored. One way to do this in the decoder 250 is to pre-mix the two uncorrelated signals d2(E1(p,k)) and d3(E1(p,k)) with a 2x2 matrix obtained by the following equation,

[0167]

number

[0168] The parameter γ is transmitted to the decoder 250 so that it can generate an uncorrelated signal, which is used to reconstruct the normal correlation γ between the original intrinsic signals E2 and E3. Alternatively, the mixing matrix G can be set to a fixed value in the decoder 250, as shown below, which generally improves the reconstruction of the correlation between E2 and E3.

[0169]

number

[0170] A parameter-based sound field coding framework may be combined with a multi-channel waveform coding framework over selected subbands of the sound field's characteristic representations to obtain a mixed coding framework. In particular, waveform coding may be performed for the low-frequency bands of E2 and E3, and parameter coding for the remaining frequency bands. Specifically, the encoder 1200 (and decoder 250) may be configured to calculate a starting band. For subbands lower than the starting band, the characteristic signals E1, E2, and E3 may be waveform coded individually. When the subbands are in the starting band, and when they are above the starting band, the characteristic signals E2 and E3 may be coded parameterally (as described herein).

[0171] Figure 24A is a flowchart of an exemplary method 1300 for encoding frames of a sound field signal 110 containing multiple audio signals (or audio channels). Method 1300 includes step 301 of calculating an energy-compressing orthogonal transform V (e.g., KLT) based on frames of the sound field signal 110. As described herein, it may be preferable to use a non-adaptive transform to transform the sound field signal 110 in an captured region (e.g., LRS region) into a sound field signal 111 in a non-adaptive transform region (e.g., WXY region). In such a case, the energy-compressing orthogonal transform V may be calculated based on the sound field signal 111 in the non-adaptive transform region. Method 300 may further include step 302 of applying the energy-compressing orthogonal transform V to frames of the sound field signal 110 (or sound field signals 111 derived from these frames). This results in frames of a rotated sound field signal 112 containing multiple rotated audio signals E1, E2, E3 (step 303). The rotated sound field signal 112 corresponds to the sound field signal 112 within the adaptive transformation region (e.g., the E1, E2, E3 region). Method 300 may include a step 304 of encoding the first rotated audio signal E1 among the multiple rotated audio signals E1, E2, E3 (e.g., using a single channel waveform encoder 103). Furthermore, Method 300 may include a step 305 of calculating a set of prediction parameters a2, b2 to calculate the second rotated audio signal E2 among the multiple rotated audio signals E1, E2, E3 based on the first rotated audio signal E1.

[0172] Figure 24B is a flowchart of an exemplary method 350 for decoding frames of a reconstructed sound field signal 117 containing multiple reconstructed audio signals from a spatial bitstream 221 and a downmixed bitstream 222.

[0173] [Table 10] This specification has described methods and systems for encoding sound field signals. In particular, it has described a parameter coding framework for sound field signals that can reduce the bitrate while maintaining a certain level of perceptual quality. Furthermore, the parameter coding framework provides high-quality downmix signals at low bitrates, which is beneficial for implementing hierarchical video conferencing systems.

[0174] Combinations of Embodiments and Application Background All embodiments and their variations discussed above may be implemented in any combination thereof, and components that are mentioned in different parts / embodiments but have the same or similar function may be implemented as the same or separate components.

[0175] For example, different embodiments and variations of the first compensation unit 400 for the monaural component PLC may be randomly combined with different embodiments and variations of the second compensation unit 600 and the second transducer 1000 for the spatial component PLC. Also, in Figures 9A and 9B, different embodiments and variations of the main compensation unit 408 for both major and minor monaural components of the unpredicted PLC may be randomly combined with different embodiments and variations of the predictive parameter calculator 412, the third compensation unit 414, the predictive decoder 410, and the adjustment unit 416 for the predictive PLC of the minor monaural component.

[0176] As discussed above, packet loss can occur anywhere along the path from the source communication terminal to the server (if any), and from there to the destination communication terminal. Therefore, the PLC device proposed herein may be applied to either a server or a communication terminal. When applied to a server as shown in Figure 12, the voice signal, after compensating for packet loss, may be repackaged by the packetization unit 900 and transmitted to the destination communication terminal. If there are multiple users conversing simultaneously (this can be determined using voice interval detection (VAD) technology), the mixer 800 needs to perform a mixing operation to combine multiple streams of speech signals into one before transmitting the speech signals of multiple users to the destination communication terminal. This may be done after the PLC operation of the PLC device, but before the packetization operation of the packetization unit 900.

[0177] When applied to a communication terminal as shown in Figure 13, a second inverse converter 700A may be provided to convert the created frame into a spatial audio signal in intermediate output format. Alternatively, as shown in Figure 14, a second decoder 700B may be provided to decode the created frame into a spatial audio signal in the time domain, such as a binaural audio signal. The other elements in Figures 12 to 14 are the same as in Figure 3, so a detailed explanation of them is omitted.

[0178] Accordingly, this specification also provides an audio processing system such as an audio communication system, which comprises a server (such as an audio conferencing mixing server) equipped with a packet loss compensation device as discussed above and / or a communication terminal equipped with a packet loss compensation device as discussed above.

[0179] It can be seen that the server and communication terminal shown in Figures 12 to 14 are located on either the destination or decoding side. This is because the PLC device provided is intended to compensate for packet loss that occurs before reaching the destination (including the server and destination communication terminal). Conversely, the second converter 1000, as considered with reference to Figure 11, is intended to be used on either the source side or the encoding side, specifically the source communication terminal or server.

[0180] Therefore, the audio processing system discussed above may further include a communication terminal as a source communication terminal, which includes a second converter 1000 for converting a spatial audio signal in input format into a frame in transmission format, each frame containing at least one monaural component and at least one spatial component.

[0181] As discussed at the beginning of the embodiments for carrying out the inventions herein, the embodiments herein may be implemented in hardware, software, or both. Figure 15 is a block diagram illustrating an exemplary system for carrying out an embodiment of this specification.

[0182] In Figure 15, the central processing unit (CPU) 801 executes various processes according to programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. RAM 803 also stores data required when the CPU 801 executes various processes, as needed.

[0183] The CPU 801, ROM 802, and RAM 803 are connected to each other via bus 804. The input / output interface 805 is also connected to bus 804. The following elements are connected to the input / output interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a display such as a cathode ray tube (CRT), liquid crystal display (LCD), and loudspeaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card and modem. The communication section 809 carries out communication processes via a network such as the Internet.

[0184] Drive 810 is also connected to the input / output interface 805 as needed. Removable media 811 such as magnetic disks, optical disks, magneto-optical disks, and semiconductor memory are mounted on drive 810 as needed, and computer programs read from there are installed in the storage section 808 as needed.

[0185] If the aforementioned components are implemented by software, the programs that make up the software are installed from a network such as the Internet or from a storage medium such as a removable medium 811.

[0186] Packet loss compensation method In the process of describing the packet loss compensation device of the above embodiment, several processes or methods will also be disclosed. Below, a summary of these methods will be given without repeating some of the details already discussed above. It should be noted that although these methods are disclosed in the process of describing the packet loss compensation device, they do not necessarily have to employ the components described, nor do they necessarily have to be implemented by such components. For example, embodiments of the packet loss compensation device may be partially or completely implemented using hardware and / or firmware, and the packet loss compensation method discussed below may also be fully implemented by a computer-executable program, although this method may employ the hardware and / or firmware of the packet loss compensation device.

[0187] According to an embodiment of the present specification, there is provided a packet loss compensation method for compensating packet loss in a stream of audio packets, wherein each audio packet includes at least one audio frame in a transmission format including at least one monaural component and at least one spatial component. In the present specification, it is proposed to perform different PLCs for different components within an audio frame. That is, in the case of a lost frame in a lost packet, one operation for creating at least one monaural component for the lost frame and another operation for creating at least one spatial component for the lost frame are executed. Here, it should be noted that the two operations do not necessarily have to be executed simultaneously for the same lost frame.

[0188] (The audio frame of the transmission format) may be encoded based on an adaptive transformation, which can transform an audio signal (in an input format such as an LRS signal or an Ambisonics B format (WXY) signal) into a monaural component and a spatial component during transmission. An example of the adaptive transformation is eigenvalue decomposition by parameters, and the monaural component may include at least one eigenchannel component, and the spatial component may include at least one spatial parameter. Other examples of the adaptive transformation may include principal component analysis (PCA). Regarding eigenvalue decomposition by parameters, an example is KLT encoding, in which a plurality of rotated audio signals as eigenchannel components and a plurality of spatial parameters can be obtained. Generally, the spatial parameter is derived from a transformation matrix for converting an audio signal in the input format into an audio frame in the transmission format, for example, for converting an Ambisonics B format audio signal into a plurality of rotated audio signals.

[0189] In the case of a spatial audio signal, the continuity of spatial parameters is extremely important. Therefore, in order to compensate for lost frames, at least one spatial component for a lost frame can be created by smoothing the values of at least one spatial component of (one or more) adjacent frames such as (one or more) past frames and / or (one or more) future frames. Another method is to create at least one spatial component for a lost frame via an interpolation algorithm based on the values of the corresponding spatial components within at least one adjacent past frame and at least one adjacent future frame. If there are multiple consecutive frames, all lost frames can be created via a single interpolation operation. Additionally, a more straightforward method is to create at least one spatial component for a lost frame by replicating the corresponding spatial component within the last frame. In the last case, in order to achieve the stability of the spatial parameters, the spatial parameters themselves can be directly smoothed, or the (elements of) transformation matrices such as the covariance matrix used to derive the spatial parameters can be smoothed so that the spatial parameters can be pre-smoothed on the encoding side.

[0190] In the case of a monaural component, if lost frames are to be compensated, the monaural component can be created by replicating the corresponding monaural component within adjacent frames. Here, adjacent frames mean the most recent or past or future frames sandwiching (one or more) other frames. In a variant, an attenuation coefficient may be used. Depending on the application background, it may not be possible to create several monaural components for a lost frame, and only at least one monaural component may be created by replication. Specifically, a monaural component such as an eigen-channel component (rotated audio signal) may comprise one main monaural component and several other less important but different monaural components. Therefore, only the main monaural component or the first two important monaural components can be replicated, but not limited to this.

[0191] Lost packets, such as those containing multiple consecutive frames, may contain multiple audio frames or multiple packets may be lost. In this context, it is reasonable to create at least one monaural component for at least one earlier lost frame by duplicating the corresponding monaural component in an adjacent past frame, with or without using an attenuation coefficient, and to create at least one monaural component for at least one later lost frame by duplicating the corresponding monaural component in an adjacent future frame, with or without using an attenuation coefficient. In other words, for the earlier (one or more) frames of a lost frame, the monaural component is created by duplicating past frames, and for the later (one or more) frames, the monaural component is created by duplicating future frames.

[0192] In addition to direct duplication, another embodiment proposes compensating for lost monaural components in the time domain. First, at least one monaural component in at least one preceding frame before the lost frame is converted into a time-domain signal, and then packet loss is compensated for in that time-domain signal to produce a time-domain signal with packet loss compensated. Finally, the time-domain signal with packet loss compensated can be converted into the form of at least one monaural component to produce a monaural component created corresponding to at least one monaural component in the lost frame. Here, if the monaural components in the audio frame are decoded in a non-overlapping framework, it is sufficient to convert only the monaural component in the last frame into the time domain. If the monaural components in the audio frame are encoded in an overlapping framework such as MDCT conversion, it is preferable to convert at least two preceding frames into the time domain.

[0193] Instead of doing this, if there are more consecutive loss frames, a more efficient bidirectional method can be used to compensate for some loss frames in the time domain PLC and some loss frames in the frequency domain. One example is where earlier loss frames are compensated in the time domain PLC, and later loss frames are compensated by simple duplication, that is, by duplicating the corresponding monaural components in adjacent (one or more) future frames. Attenuation coefficients may or may not be used for duplication.

[0194] To improve coding efficiency and bitrate, parameter coding / predictive coding may be employed, in which case each audio frame in the audio stream further includes, in addition to spatial parameters and at least one monaural component (generally the primary monaural component), at least one predictive parameter used to predict at least one other monaural component for that frame based on at least one monaural component in the frame. For such an audio stream, PLC may also be performed on (one or more) predictive parameters. As shown in Figure 16, for a lossy frame, at least one monaural component (generally the primary monaural component) that is to be transmitted is created by any existing method or by methods considered above, including time-domain PLC, bidirectional PLC, or replication with or without attenuation coefficients (operation 1602). In addition, (one or more) predictive parameters can be created to predict (one or more) other monaural components (generally (one or more) less important monaural components) based on the primary monaural component (operation 1604).

[0195] Prediction parameters can be created in a similar manner to spatial parameters, for example, by duplicating the corresponding prediction parameter in the last frame, or by smoothing the values ​​of the corresponding prediction parameter in (one or more) adjacent frames, or by interpolation using the values ​​of the corresponding prediction parameter in past and future frames, with or without using a decay coefficient. For prediction PLCs of independently encoded audio streams (Figures 18-21), the creation operation may be carried out similarly.

[0196] The created primary monaural component and prediction parameters can be used to predict other monaural components based on them (operation 1608), and the created primary monaural component and the other (one or more) predicted monaural components (along with spatial parameters) constitute the created frame concealment packet / frame loss. However, prediction operation 1608 does not necessarily have to be performed immediately after creation operations 1602 and 1604. If mixing is not required within the server, the created primary monaural component and the created prediction parameters may be forwarded directly to the destination communication terminal, in which case prediction operation 1608 and (one or more) further operations are performed.

[0197] The prediction operation in predictive PLC is the same as the prediction operation in predictive coding (even if predictive PLC is performed on an unpredicted / independently coded audio stream). That is, at least one other monaural component of a lost frame may be predicted based on one monaural component and its uncorrelated version, using at least one prediction parameter created with or without an attenuation coefficient. For example, a monaural component in a past frame corresponding to one monaural component created for a lost frame may be considered an uncorrelated version of the created monaural component. In the case of predictive PLC for independently coded audio streams (Figures 18-21), the prediction operation may be performed similarly.

[0198] Predictive PLC may also be applied to unpredictable / independently coded audio streams, in which case each audio frame has at least two monaural components, generally a primary monaural component and at least one less important monaural component. Predictive PLC predicts the less important monaural component based on the primary monaural component already created to compensate for the lost frame, using a method similar to predictive coding as discussed above. In the case of independently coded audio streams, there are no available prediction parameters because they are within the PLC and cannot be calculated from the current frame (because the current frame is lost and needs to be created / reconstructed). Therefore, prediction parameters may be derived from past frames, regardless of whether those past frames were transmitted successfully or created / reconstructed for the PLC. Next, in one embodiment as shown in Figure 17, creating at least one monaural component includes creating one of at least two monaural components for a loss frame (operation 1602), calculating at least one prediction parameter for the loss frame using past frames (operation 1606), and using the created at least one prediction parameter to predict the other at least one of the at least two monaural components of the loss frame based on the created monaural component (operation 1608).

[0199] In the case of independently encoded audio streams, if predictive PLC is always performed for each lost frame, efficiency can be low, especially when there are a relatively large number of lost packets. In this context, predictive PLC for independently encoded audio streams can be combined with normal PLC for predictively encoded audio streams. That is, once the predictive parameters have been calculated for earlier lost frames, subsequent lost frames can utilize the calculated predictive parameters through normal PLC operations such as duplication, smoothing, and interpolation, as discussed above.

[0200] Therefore, as shown in Figure 18, in the case of multiple consecutive loss frames, for the first loss frame (operation 1603, "Y"), then, based on the last frame (which was transmitted successfully), prediction parameters are calculated (operation 1606) and used to predict the other monaural components (operation 1608). Starting from the second loss frame, a normal PLC can be performed to create a predictive instrument using the prediction parameters calculated for the first loss frame (see dashed arrow in Figure 18) (operation 1604).

[0201] More generally, an adaptive PLC method can be proposed, which can be adapted and used in either a predictive coding framework or a non-predictive / independent coding framework. For the first loss frame in an independent coding framework, predictive PLC is performed, but for subsequent (one or more) loss frames in an independent coding framework, or for a predictive coding framework, normal PLC is performed. Specifically, as shown in Figure 19, for any loss frame, at least one monaural component, such as the primary monaural component, may be created using any of the PLC methods discussed above (operation 1602). Other generally less important monaural components may be created / restored in different ways. If at least one prediction parameter is present in the last frame prior to the loss frame (the “predictive coding” branch in operation 1601), or if at least one prediction parameter has been calculated for the last frame prior to the loss frame (meaning the last frame is also a loss frame, but its prediction parameter was calculated in operation 1606), or if at least one prediction parameter has been created for the last frame prior to the loss frame (meaning the last frame is also a loss frame, but its prediction parameter was created in operation 1604), then at least one prediction parameter for the current loss frame may be created via the normal PLC method based on at least one prediction parameter for the last frame (operation 1604). In that case, at least one prediction parameter can be calculated for the loss frame using previous frames only if the last frame prior to the loss frame does not contain a prediction parameter (the “non-predictive coding” branch in operation 1601) and no prediction parameter has been created / calculated for the last frame prior to the loss frame, i.e., if the loss frame is the first of several consecutive loss frames (“Y” in operation 1603) (operation 1606).Next, the other mono component of at least one of the two mono components of the loss frame may be predicted based on one mono component created (from operation 1602) using at least one prediction parameter calculated (from operation 1606) or at least one prediction parameter created (from operation 1604) (operation 1608).

[0202] In a modified example, for independently encoded audio streams, the prediction PLC can be combined with a normal PLC to further randomize the results and compensate for packet loss, resulting in a more natural-sounding audio stream. Next, as shown in Figure 20 (corresponding to Figure 18), both the prediction operation 1608 and the creation operation 1609 are performed, and the results are combined (operation 1612) to obtain the final result. Combination operation 1612 can be considered as an operation that adjusts one to the other in any way. For example, the adjustment operation may include calculating a weighted average of at least one predicted other mono component and at least one created other mono component as the final result of at least one other mono component. The weight coefficients may be calculated depending on which of the prediction result and the creation result is dominant, and according to the specific application context. In the embodiment described with reference to Figure 19, combination operation 1612 may be added as shown in Figure 21, but a detailed explanation is omitted here. In fact, combination operation 1612 is also possible for the solution shown in Figure 17, but this is not illustrated.

[0203] The calculation of (one or more) prediction parameters is similar to that of the prediction / parameter coding process. In the prediction coding process, (one or more) prediction parameters for the current frame may be calculated based on the first rotated audio signal (E1) (major monaural component) and at least a second rotated audio signal (E2) (at least one less important monaural component) of the same frame (Equations (19) and (20)). Specifically, the prediction parameters may be calculated such that the mean squared error of the prediction residual between the second rotated audio signal (E2) (at least one less important monaural component) and the correlated component of the second rotated audio signal (E2) is small. The prediction parameters may further include an energy adjustment gain, which may be calculated based on the ratio of the amplitude of the prediction residual to the amplitude of the first rotated audio signal (E1) (major monaural component). In a modified version, this calculation may be based on the ratio of the root mean square of the predicted residual to the root mean square of the initially rotated audio signal (E1) (main monaural component) ((Equations (21) and (22))). To avoid abrupt fluctuations in the calculated energy adjustment gain, a Ducker adjustment operation can be applied, which includes calculating an uncorrelated signal based on the initially rotated audio signal (E1) (main monaural component), calculating a second energy index of the uncorrelated signal and a first energy index of the initially rotated audio signal (E1) (main monaural component), and calculating the energy adjustment gain based on the uncorrelated signal if the second index is greater than the first index (Equations (26) to (37)).

[0204] In predictive PLC, the calculation of (one or more) predictive parameters is similar, the difference being in the current frame (loss frame), while (one or more) predictive parameters are calculated based on (one or more) previous frames. In other words, (one or more) predictive parameters are calculated for the last frame prior to the loss frame and then used to compensate for the loss frame.

[0205] Therefore, in a predictive PLC, at least one predictive parameter for a loss frame may be calculated based on the monaural component in the last frame prior to the loss frame that corresponds to one monaural component created for the loss frame, and the monaural component in the last frame that corresponds to the monaural component that is to be predicted for the loss frame (Equation (9)). Specifically, at least one predictive parameter for a loss frame may be calculated such that the mean squared error of the prediction residual between the monaural component in the last frame that corresponds to the monaural component that is to be predicted for the loss frame and its correlated component is small.

[0206] At least one prediction parameter may further include an energy adjustment gain, which may be calculated based on the ratio of the amplitude of the prediction residual to the amplitude of the monaural component in the last frame prior to the loss frame, corresponding to one monaural component created for the loss frame. In a modified example, the second energy adjustment gain may be calculated based on the ratio of the root mean square of the prediction residual to the root mean square of the monaural component in the last frame prior to the loss frame, corresponding to one monaural component created for the loss frame (Equation (10)).

[0207] To prevent the energy adjustment gain from fluctuating too rapidly, a Ducker algorithm may be implemented (Equations (11) and (12)). This involves calculating an uncorrelated signal based on the monaural component in the last frame prior to the loss frame, corresponding to one monaural component created for the loss frame; calculating a second energy index of the uncorrelated signal and a first energy index of the monaural component in the last frame prior to the loss frame, corresponding to one monaural component created for the loss frame; and calculating a second energy adjustment gain based on the uncorrelated signal if the second index is greater than the first index.

[0208] After PLC, a new packet is created to replace the lost packet. Next, the created packet, along with the successfully transmitted voice packet, may undergo retro-adaptive transformation to convert it into an inversely transformed sound field signal such as a WXY signal. One example of retro-adaptive transformation may be the inverse Karhunen-Loeve (KLT) transform.

[0209] Similar to the embodiment of the packet loss compensation device, any combination of the embodiment of the PLC method and its variations is possible. The methods and systems described herein may be implemented as software, firmware, and / or hardware. Certain elements may be implemented, for example, as software running on a digital signal processor or microprocessor. Other elements may be implemented, for example, as hardware and / or as application-specific integrated circuits. Signals found in the methods and systems described herein may be stored in a medium such as random-access memory or optical storage media. Signals may be transmitted over a network such as a radio network, satellite network, wireless network, or wired network, such as the Internet. Typical devices utilizing the methods and systems described herein are portable electronic devices or other consumer devices used to store and / or render audio signals.

[0210] Note that the terms used in this specification are for the sole purpose of describing particular embodiments and are not intended to limit the specification. As used in this specification, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. The terms "comprises" and / or "comprising", when used in this specification, specify the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or components, and / or groups thereof.

[0211] In addition to corresponding structures, materials, acts, and equivalents of all means or steps, the functional elements in the following claims are intended to include any structure, material, or act for performing the functions thereof in combination with the other claimed elements specifically recited. The description of the present specification is presented for purposes of illustration and description and is not intended to be exhaustive or to limit the application to the forms disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the spirit and scope of the present specification. Embodiments are chosen and described in order to best explain the principles and practical applications of the present specification, and to enable others of ordinary skill in the art to understand the present specification for various embodiments with various modifications as are suited to the particular use contemplated. The technical ideas that can be grasped from the above embodiments are described below. (Appendix 1) A packet loss compensation device for compensating for packet loss in a stream of voice packets, wherein each voice packet includes at least one voice frame in a transmission format including at least one monaural component and at least one spatial component, in the packet loss compensation device, A first compensation unit for creating the at least one monaural component for a loss frame of a lost packet, A packet loss compensation device comprising a second compensation unit for creating at least one spatial component for the loss frame. (Note 2) The packet loss compensation device described in Appendix 1, wherein the aforementioned audio frame is encoded based on an adaptive orthogonal transform. (Note 3) The aforementioned audio frame is encoded based on parameter-based intrinsic decomposition, The aforementioned at least one monaural component includes at least one intrinsic channel component, The packet loss compensation device according to Appendix 1, wherein the at least one spatial component includes at least one spatial parameter. (Note 4) The packet loss compensation device according to any one of the appendices 1 to 3, wherein the first compensation unit is configured to create the at least one monaural component for the lost frame by duplicating the corresponding monaural component in an adjacent frame, with or without using an attenuation coefficient. (Note 5) At least two consecutive frames are lost. The packet loss compensation device according to any one of the appendices 1 to 4, wherein the first compensation unit is configured to create the at least one monaural component for at least one earlier loss frame by replicating the corresponding monaural component in an adjacent past frame with or without using an attenuation coefficient, and to create the at least one monaural component for at least one later loss frame by replicating the corresponding monaural component in an adjacent future frame with or without using an attenuation coefficient. (Note 6) The first compensation unit described above is: A first converter for converting the at least one monaural component in at least one past frame prior to the loss frame into a time-domain signal, A time-domain compensation unit for compensating for the packet loss in the time-domain signal to obtain a time-domain signal with packet loss compensation, The packet loss compensation device according to Appendix 1, further comprising a first inverse converter for converting the time-domain signal with the packet loss compensated into the form of the at least one monaural component to a created monaural component corresponding to the at least one monaural component in the loss frame. (Note 7) At least two consecutive frames are lost. The packet loss compensation device according to Appendix 6, wherein the first compensation unit is further configured to create the at least one monaural component for at least one later loss frame by duplicating the corresponding monaural component in an adjacent future frame, with or without using an attenuation coefficient. (Note 8) Each audio frame further comprises at least one prediction parameter used for prediction based on the at least one mono component within the audio frame and at least one other mono component within the audio frame. The first compensation unit described above is: A main compensation unit for creating the at least one monaural component for the loss frame, A packet loss compensation device according to any one of the appendices 1 to 7, further comprising a third compensation unit for creating the at least one prediction parameter for the lost frame. (Note 9) The packet loss compensation device according to Appendix 8, wherein the third compensation unit is configured to create the at least one prediction parameter for the lost frame by duplicating the corresponding prediction parameter in the last frame, by smoothing the values ​​of the corresponding prediction parameter in one or more adjacent frames, or by interpolation using the values ​​of the corresponding prediction parameter in past and future frames, with or without using an attenuation coefficient. (Note 10) The packet loss compensation device according to Appendix 8, further comprising a predictive decoder for predicting the at least one other monaural component for the loss frame based on one monaural component created using at least one predictive parameter created. (Note 11) The packet loss compensation device according to Appendix 10, wherein the predictive decoder is configured to predict the at least one other monaural component for the loss frame based on one monaural component and its uncorrelated version, using at least one predictive parameter created, with or without an attenuation coefficient. (Note 12) The packet loss compensation device according to Appendix 11, wherein the predictive decoder is configured to take in the monaural component in past frames corresponding to the monaural component created for the loss frame as the uncorrelated version of the monaural component created. (Note 13) Each audio frame contains at least two mono components. The first compensation unit described above is: A main compensation unit for creating one of the at least two monaural components from the loss frame, A predictor parameter calculator for calculating at least one predictor parameter for the loss frame using past frames, A packet loss compensation device according to any one of the appendices 1 to 7, comprising a predictive decoder for predicting the other mono component of the at least two mono components of the loss frame based on one mono component created using at least one predictive parameter created. (Note 14) The first compensation unit described above is: If at least one prediction parameter is included in the last frame prior to the loss frame or is created and / or calculated for that last frame, the system further comprises a third compensation unit for creating the at least one prediction parameter for the loss frame based on the at least one prediction parameter for the last frame, The predictive parameter calculator is configured to calculate at least one predictive parameter for the loss frame using the previous frame if the predictive parameter is not included or has not been created or calculated for the last frame prior to the loss frame. The packet loss compensation device according to Appendix 13, wherein the predictive decoder is configured to predict, based on one monaural component created, the other monaural component of at least one of at least two monaural components of the loss frame, using at least one predictive parameter that has been calculated or created. (Note 15) The main compensation unit is further configured to create the at least one other mono component, The packet loss compensation apparatus according to Appendix 13, wherein the first compensation unit further includes an adjustment unit for adjusting the at least one other monaural component predicted by the predictive decoder with the at least one other monaural component created by the main compensation unit. (Note 16) The packet loss compensation device according to Appendix 15, wherein the adjustment unit is configured to calculate a weighted average of the at least one other monaural component predicted by the predictive decoder and the at least one other monaural component created by the main compensation unit as the final result of the at least one other monaural component. (Note 17) The packet loss compensation device according to Appendix 14, wherein the third compensation unit is configured to create the at least one prediction parameter for the lost frame by duplicating the corresponding prediction parameter in the last frame, or by smoothing the values ​​of the corresponding prediction parameter in one or more adjacent frames, or by interpolation using the values ​​of the corresponding prediction parameter in past and future frames, with or without using an attenuation coefficient. (Note 18) The packet loss compensation device according to Appendix 13, wherein the predictive decoder is configured to predict the at least one other monaural component of the loss frame based on one monaural component and its uncorrelated version, using at least one predictive parameter created, with or without using an attenuation coefficient. (Note 19) The packet loss compensation device according to Appendix 18, wherein the predictive decoder is configured to take in the monaural component in past frames corresponding to the monaural component created for the loss frame as the uncorrelated version of the monaural component created. (Note 20) The packet loss compensation device according to Appendix 13, wherein the prediction parameter calculator is configured to calculate the at least one prediction parameter for the loss frame based on the monaural component in the last frame prior to the loss frame that corresponds to one monaural component created for the loss frame, and the monaural component in the last frame that corresponds to the monaural component that is predicted for the loss frame. (Note 21) The packet loss compensation device according to Appendix 20, wherein the prediction parameter calculator is configured to calculate the at least one prediction parameter for the loss frame such that the mean squared error of the prediction residual between the monaural component in the last frame corresponding to the monaural component that is to be predicted for the loss frame and its correlated component is small. (Note 22) The at least one prediction parameter includes an energy adjustment gain, The packet loss compensation device according to Appendix 21, wherein the prediction parameter calculator is configured to calculate the energy adjustment gain based on the ratio of the amplitude of the prediction residual to the amplitude of the monaural component in the last frame prior to the loss frame, which corresponds to one monaural component created for the loss frame. (Note 23) The packet loss compensation device according to Appendix 22, wherein the prediction parameter calculator is configured to calculate the energy adjustment gain based on the ratio of the root mean square of the prediction residual to the root mean square of the monaural component in the last frame prior to the loss frame, which corresponds to one monaural component created for the loss frame. (Note 24) The at least one prediction parameter includes an energy adjustment gain, The aforementioned predictive parameter calculator is: An uncorrelated signal is calculated based on the mono component in the last frame prior to the loss frame, which corresponds to one mono component created for the loss frame. A second index of the energy of the uncorrelated signal and a first index of the energy of the monaural component in the last frame prior to the loss frame, corresponding to one monaural component created for the loss frame, are calculated. The packet loss compensation device according to Appendix 20, configured to calculate the energy adjustment gain based on the uncorrelated signal when the second index is greater than the first index. (Note 25) The packet loss compensation device according to Appendix 1, wherein the second compensation unit is configured to create the at least one spatial component for the loss frame by smoothing the values ​​of the at least one spatial component of one or more adjacent frames. (Note 26) The packet loss compensation device according to Appendix 1, wherein the second compensation unit is configured to create the at least one spatial component for the loss frame via an interpolation algorithm based on the values ​​of the corresponding spatial components in at least one adjacent past frame and at least one adjacent future frame. (Note 27) At least two consecutive frames are lost. The packet loss compensation device according to Appendix 25 or 26, wherein the second compensation unit is configured to create the at least one spatial component for all of the lost frames based on the values ​​of the corresponding spatial components in at least one adjacent past frame and at least one adjacent future frame. (Note 28) The packet loss compensation device according to Appendix 1, wherein the second compensation unit is configured to create the at least one spatial component for the lost frame by replicating the corresponding spatial component in the last frame. (Note 29) A packet loss compensation method for compensating for packet loss within a stream of voice packets, wherein each voice packet includes at least one voice frame in a transmission format that includes at least one monaural component and at least one spatial component, Creating at least one monaural component for the lost frame of the lost packet, A packet loss compensation method comprising creating the at least one spatial component for the loss frame. (Note 30) The packet loss compensation method described in Appendix 29, wherein the aforementioned audio frame is encoded based on an adaptive orthogonal transform. (Note 31) The aforementioned audio frame is encoded based on parameter-based intrinsic decomposition, The aforementioned at least one monaural component includes at least one intrinsic channel component, The packet loss compensation method according to Appendix 29, wherein the at least one spatial component includes at least one spatial parameter. (Note 32) The packet loss compensation method according to any one of the appendices 29 to 31, wherein creating the at least one monaural component includes creating the at least one monaural component for the lost frame by duplicating a corresponding monaural component in an adjacent frame, with or without using an attenuation coefficient. (Note 33) A packet loss compensation method according to any one of the appendices 29 to 32, wherein at least two consecutive frames are lost, and creating the at least one monaural component includes creating the at least one monaural component for at least one earlier lost frame by duplicating a corresponding monaural component in an adjacent past frame, with or without using an attenuation coefficient, and creating the at least one monaural component for at least one later lost frame by duplicating a corresponding monaural component in an adjacent future frame, with or without using an attenuation coefficient. (Note 34) Creating the aforementioned at least one monaural component is Converting the at least one monaural component in at least one past frame prior to the loss frame into a time-domain signal, To compensate for the packet loss in the aforementioned time-domain signal and obtain a time-domain signal with packet loss compensation, The packet loss compensation method according to Appendix 29, comprising converting the time-domain signal with the packet loss compensated into the form of the at least one monaural component to obtain a created monaural component corresponding to the at least one monaural component in the loss frame. (Note 35) The packet loss compensation method according to Appendix 34, wherein at least two consecutive frames are lost, and creating the at least one monaural component further comprises creating the at least one monaural component for at least one later lost frame by duplicating a corresponding monaural component in an adjacent future frame, with or without using an attenuation coefficient. (Note 36) Each audio frame further comprises at least one prediction parameter used for prediction based on the at least one mono component within the audio frame and at least one other mono component within the audio frame. Creating the aforementioned at least one monaural component is To create at least one monaural component from the loss frame, A packet loss compensation method according to any one of the appendices 29 to 35, comprising creating the at least one prediction parameter for the lost frame. (Note 37) The packet loss compensation method according to Appendix 36, wherein creating the at least one prediction parameter includes creating the at least one prediction parameter for the lost frame by duplicating the corresponding prediction parameter in the last frame, or by smoothing the values ​​of the corresponding prediction parameter in one or more adjacent frames, or by interpolation using the values ​​of the corresponding prediction parameter in past and future frames, with or without using an attenuation coefficient. (Note 38) The packet loss compensation method according to Appendix 36, further comprising predicting the at least one other monaural component for the loss frame based on the one monaural component created using the at least one prediction parameter created. (Note 39) The packet loss compensation method according to Appendix 38, wherein the predicted behavior includes predicting the at least one other monaural component for the loss frame from one monaural component and its uncorrelated version, using at least one predicted parameter created, with or without using an attenuation coefficient. (Note 40) The packet loss compensation method according to Appendix 39, wherein the predicted action is to take the monaural component in a past frame corresponding to the monaural component created for the loss frame as the uncorrelated version of the monaural component created. (Note 41) Each audio frame contains at least two mono components. Creating the aforementioned at least one monaural component is equivalent to creating one of the aforementioned at least two monaural components for the loss frame. Calculate at least one prediction parameter for the loss frame using past frames. A packet loss compensation method according to any one of the appendices 29 to 35, comprising predicting the other monaural component of at least one of the at least two monaural components of the lost frame based on one monaural component created using at least one prediction parameter created. (Note 42) Creating the aforementioned at least one monaural component is If at least one prediction parameter is included in the last frame prior to the loss frame or is created and / or calculated for that last frame, the method further includes creating the at least one prediction parameter for the loss frame based on the at least one prediction parameter for the last frame. The calculation operation includes calculating the at least one predictive parameter for the loss frame using the previous frame if the predictive parameter is not included or has not been created or calculated for the last frame prior to the loss frame. The packet loss compensation method according to Appendix 41, wherein the prediction operation includes predicting the other monaural component of at least one of the at least two monaural components of the lost frame based on one monaural component created using the calculated or created prediction parameter. (Note 43) To create at least one other mono component, The packet loss compensation method according to Appendix 41, further comprising coordinating the at least one other mono component predicted by the predictive operation with the at least one other mono component created. (Note 44) The packet loss compensation method according to Appendix 43, wherein the adjustment operation includes calculating a weighted average of the predicted at least one other mono component and the created at least one other mono component as the final result of the at least one other mono component. (Note 45) The packet loss compensation method according to Appendix 42, wherein creating the at least one prediction parameter includes creating the at least one prediction parameter for the lost frame by duplicating the corresponding prediction parameter in the last frame, or by smoothing the values ​​of the corresponding prediction parameter in one or more adjacent frames, or by interpolation using the values ​​of the corresponding prediction parameter in past and future frames, with or without using an attenuation coefficient. (Note 46) The packet loss compensation method according to Appendix 41, wherein the prediction operation includes predicting the at least one other monaural component of the lost frame based on one monaural component and its uncorrelated version, using at least one prediction parameter created, with or without using an attenuation coefficient. (Note 47) The packet loss compensation method according to Appendix 46, wherein the predictive operation involves taking in a monaural component in a past frame corresponding to the monaural component created for the loss frame as the uncorrelated version of the monaural component created. (Note 48) The packet loss compensation method according to Appendix 41, wherein the calculation operation includes calculating the at least one prediction parameter for the loss frame based on a monaural component in the last frame prior to the loss frame that corresponds to one monaural component created for the loss frame, and a monaural component in the last frame that corresponds to the monaural component that is predicted for the loss frame. (Note 49) The packet loss compensation method according to Appendix 48, wherein the calculation operation includes calculating the at least one prediction parameter for the loss frame such that the mean squared error of the predicted residual between the monaural component in the last frame corresponding to the monaural component that is to be predicted for the loss frame and its correlated component is small. (Note 50) The at least one prediction parameter includes an energy adjustment gain, The packet loss compensation method according to Appendix 49, wherein the calculation operation includes calculating the energy adjustment gain based on the ratio of the amplitude of the predicted residual to the amplitude of a monaural component in the last frame prior to the lost frame, which corresponds to one monaural component created for the lost frame. (Note 51) The packet loss compensation method according to Appendix 50, wherein the calculation operation includes calculating the energy adjustment gain based on the ratio of the root mean square of the predicted residual to the root mean square of the monaural component in the last frame prior to the loss frame, which corresponds to one monaural component created for the loss frame. (Note 52) The at least one prediction parameter includes an energy adjustment gain, The calculation process is as follows: To calculate an uncorrelated signal based on the mono component in the last frame prior to the loss frame, corresponding to one mono component created for the loss frame, To calculate a second index of the energy of the uncorrelated signal and a first index of the energy of the monaural component in the last frame prior to the loss frame, corresponding to one monaural component created for the loss frame, The packet loss compensation method according to Appendix 48, further comprising calculating the energy adjustment gain based on the uncorrelated signal when the second indicator is greater than the first indicator. (Note 53) The packet loss compensation method according to Appendix 29, wherein creating the at least one spatial component includes creating the at least one spatial component for the lost frame by smoothing the values ​​of the at least one spatial component for one or more adjacent frames. (Note 54) The packet loss compensation method according to Appendix 29, wherein creating the at least one spatial component includes creating the at least one spatial component for the lost frame via an interpolation algorithm based on the values ​​of the corresponding spatial components in at least one adjacent past frame and at least one adjacent future frame. (Note 55) A packet loss compensation method according to Appendix 53 or 54, wherein at least two consecutive frames are lost, and creating the at least one spatial component includes creating the at least one spatial component for all of the lost frames based on the values ​​of the corresponding spatial components in at least one adjacent past frame and at least one adjacent future frame. (Note 56) The packet loss compensation method according to Appendix 29, wherein creating the at least one spatial component includes creating the at least one spatial component for the lost frame by replicating the corresponding spatial component in the last frame. (Note 57) The calculation operation includes calculating the prediction parameters based on the following formula:

number

number

number

number

number

number

number

number

Claims

1. A packet loss compensation device for compensating for packet loss within a stream of voice packets, wherein each voice packet includes at least one voice frame in a transmission format that includes at least two monaural components and at least one spatial component. The packet loss compensation device is A first compensation unit that creates at least two monaural components for the lost frame of a lost packet, A second compensation unit that creates the at least one spatial component for the loss frame by replicating the corresponding spatial component in the last frame of one or more adjacent frames, or by an interpolation algorithm based on the values ​​of the corresponding spatial components in at least one adjacent past frame and at least one adjacent future frame, The first compensation unit is, A main compensation unit that creates at least one of the at least two monaural components for the loss frame, A predictor parameter calculator that calculates at least one predictor parameter for the loss frame using past frames, A packet loss compensation device comprising: a predictive decoder that predicts each of the remaining mono components of the at least two mono components of the loss frame based on the at least one mono component created; and the at least one predictive parameter.

2. The packet loss compensation device according to claim 1, wherein the first compensation unit is configured to create the at least one monaural component for the loss frame by duplicating the at least one monaural component of one or more adjacent frames.

3. The packet loss compensation apparatus according to claim 1 or 2, wherein the first compensation unit further includes an adjustment unit that adjusts at least one remaining monaural component predicted by the predictive decoder with at least one remaining monaural component created by the main compensation unit.

4. The packet loss compensation device according to claim 3, wherein the adjustment unit is configured to calculate a weighted average value of at least one remaining monaural component predicted by the predictive decoder and at least one remaining monaural component created by the main compensation unit.

5. A packet loss compensation method for compensating for packet loss within a stream of voice packets, wherein each voice packet includes at least one voice frame in a transmission format that includes at least two monaural components and at least one spatial component, The packet loss compensation method described above is: Creating at least two monaural components from the lost frame of the lost packet, The method comprises creating the at least one spatial component for the loss frame by replicating the corresponding spatial component in the last frame of one or more adjacent frames, or by an interpolation algorithm based on the values ​​of the corresponding spatial components in at least one adjacent past frame and at least one adjacent future frame, Creating the aforementioned two mono components is Creating at least one of the at least two monaural components for the loss frame, Calculating at least one prediction parameter for the loss frame using past frames, A packet loss compensation method comprising predicting each of the remaining mono components of the at least two mono components of the lost frame based on the at least one mono component created and the at least one prediction parameter.

6. A computer-readable medium that stores a plurality of computer program instructions that, when executed by one or more processors, cause one or more processors to perform the operation described in claim 5.

7. A packet loss compensation device for compensating for packet loss within a stream of voice packets, wherein each voice packet represents at least one voice frame containing at least two monaural components and at least one spatial component, The packet loss compensation device is A first compensation unit that creates at least two monaural components for the lost frame of a lost packet, A second compensation unit that creates the at least one spatial component for the loss frame by replicating the corresponding spatial component in the last frame of one or more adjacent frames, or by an interpolation algorithm based on the values ​​of the corresponding spatial components in at least one adjacent past frame and at least one adjacent future frame, The first compensation unit is, A main compensation unit that creates at least one of the at least two monaural components for the loss frame, A predictor parameter calculator that calculates at least one predictor parameter for the loss frame using past frames, A packet loss compensation device comprising: a predictive decoder that predicts each of the remaining mono components of the at least two mono components of the loss frame based on the at least one mono component created; and the at least one predictive parameter.

8. A packet loss compensation device for compensating for packet loss within a stream of voice packets, wherein each voice packet represents at least one frame of spatial voice using a transmission format that includes at least two monaural components and at least one spatial component, The packet loss compensation device is A first compensation unit that creates the at least two monaural components of the transmission format for the lost frame of the lost packet, A second compensation unit creates the at least one spatial component of the transmission format for the loss frame by replicating the corresponding spatial component in the last frame of one or more adjacent frames, or by an interpolation algorithm based on the values ​​of the corresponding spatial components in at least one adjacent past frame and at least one adjacent future frame, The first compensation unit is, A main compensation unit that creates at least one of the at least two monaural components of the transmission format for the loss frame, A predictor parameter calculator that calculates at least one predictor parameter for the loss frame using past frames, A packet loss compensation device comprising: a predictive decoder that predicts each of the remaining mono components of the at least two mono components of the transmission format of the loss frame based on the at least one mono component created and the at least one predictive parameter.

9. The packet loss compensation device according to claim 8, further comprising a combiner that combines the at least two monaural components created by the first compensation unit with the at least one spatial component created by the second compensation unit in order to create a replacement frame in place of at least one audio frame.

Images