Method, apparatus, and computer program for acoustic feedback suppression (AHS)

Abstract

A method and apparatus comprising computer code configured to cause one or more processors to: receive an audio signal obtained from a microphone; input the audio signal to a neural network-based AHS model, the neural network-based AHS model being trained using a training audio signal; and output an AHS signal from the neural network-based AHS model to which AHS has been applied, the AHS signal being a version of the audio signal in which acoustic feedback noise of the audio signal is suppressed and the target audio of the audio signal is sustained.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Patent Application No. 18 / 327,418, filed on 1 June 2023, which is expressly incorporated herein by reference in its entirety. [Background technology]

[0002] This disclosure relates to a set of acoustic feedback suppression (AHS) technologies for advanced audio techniques.

[0003] Acoustic feedback is a significant problem in video / audio conferencing and audio amplification systems.

[0004] Howling can occur due to coupling between a microphone and a speaker, such as when positive feedback exists between the microphone and the speaker. Specifically, a microphone signal from a microphone in an audio system may be played back through a speaker placed in the same space and then picked up again by the same microphone, forming a closed acoustic loop.

[0005] If not handled properly, this playback signal can be repeatedly looped back, potentially producing a high-pitched sound at frequencies with a loop gain of 1x or greater. This phenomenon is known as feedback.

[0006] Feedback is a significant problem for video / audio conferencing and acoustic amplification systems such as hearing aids and karaoke machines. It is not only harmful to our auditory system but also damages the amplification equipment. Therefore, feedback reduction is a critical issue in video / audio conferencing, hearing aids, karaoke, and other acoustic amplification systems.

[0007] To address this problem, many AHS solutions have been proposed, including gain control, notch filtering (NF), and adaptive feedback cancellation (AFC). Gain reduction methods can be achieved by manually lowering the volume of the amplifier or by changing the position of the audio device. However, such methods have limited applications and are not suitable for scenarios requiring high acoustic amplification. NF methods attenuate howling by adjusting their filter coefficients, creating a null at the frequencies where howling occurs. However, NF methods require accurate detection of howling and inherently distort the target sound, even capturing unexpected howling frequencies. AFC attenuates howling by using adaptive filters to estimate the acoustic path between the speaker and microphone. Because the target signal and playback signal are highly correlated, AFC methods often require correlation rejection techniques, which inevitably distorts the audio quality. [Overview of the project] [Problems that the invention aims to solve]

[0008] Therefore, for any of these reasons, a technical solution to such problems arising in computer audio technology is desired. [Means for solving the problem]

[0009] The present invention includes a method and apparatus comprising: a memory configured to store computer program code; and one or more processors configured to access the computer program code and to operate as instructed by the computer program code. The computer program is configured to cause the processors to implement: a receiving code configured to cause at least one processor to receive an audio signal acquired from a microphone; and an input code configured to cause at least one processor to input the audio signal to a neural network-based AHS model, the neural network-based AHS model being trained using a training audio signal; and an output code configured to cause at least one processor to output an AHS signal from the neural network-based AHS model to which the AHS is applied, the AHS signal being a version of the audio signal in which acoustic feedback noise is suppressed and the target audio of the audio signal is sustained.

[0010] According to an exemplary embodiment, a neural network-based AHS model is trained with a loss function that includes a combination of the scale-invariant signal-to-distortion ratio (SI-SDR) in the time domain and the mean absolute error (MAE) of the spectral amplitude in the frequency domain.

[0011] According to an exemplary embodiment, a neural network-based AHS model is trained through supervised learning.

[0012] According to an exemplary embodiment, a neural network-based AHS model comprises a first gated recurrent unit (GRU) layer configured to apply estimates to an audio signal.

[0013] According to an exemplary embodiment, the first GRU layer comprises 257 hidden units and two one-dimensional (1D) convolutional layers.

[0014] According to an exemplary embodiment, the neural network-based AHS model further comprises a second GRU layer configured to receive the output of the first GRU and to generate a covariance matrix of the acoustic howling noise and the target voice.

[0015] According to an exemplary embodiment, the second GRU layer is configured to receive both the voice signal and the output from the first GRU.

[0016] According to an exemplary embodiment, the neural network-based AHS model further comprises an enhancement filter estimation layer comprising a self-attention recurrent neural network (RNN) configured to provide an acoustic enhancement filter to an input channel of the voice signal.

[0017] According to an exemplary embodiment, the input to the first GRU layer includes the voice signal, the normalized log power spectrum (LPS) of the voice signal, the temporal correlation of the voice signal, the frequency correlation of the voice signal, and the channel covariance of the voice signal.

[0018] According to an exemplary embodiment, the input to the first GRU layer includes the concatenation of the temporal correlation of the voice signal, the frequency correlation of the voice signal, and the channel covariance of the voice signal.

[0019] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Brief Description of the Drawings

[0020] [Figure 1] It is a schematic diagram of a diagram according to an embodiment. [Figure 2] It is a simplified block diagram according to an embodiment. [Figure 3] It is a simplified diagram according to an embodiment. [Figure 4] It is a simplified diagram according to an embodiment. [Figure 5]This is a simplified diagram of an embodiment. [Figure 6] This is a simplified diagram of an embodiment. [Figure 7] This is a simplified diagram of an embodiment. [Figure 8] This is a simplified flowchart based on an embodiment. [Figure 9] This is a simplified flowchart based on an embodiment. [Figure 10] This is a simplified diagram of an embodiment. [Figure 11] This is a simplified flowchart based on an embodiment. [Figure 12] This is a simplified diagram of an embodiment. [Figure 13] This is a simplified diagram of an embodiment. [Modes for carrying out the invention]

[0021] The proposed features described below may be used separately or combined in any order. Furthermore, embodiments may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-temporary computer-readable medium.

[0022] Figure 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may code video data at its local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the coded video data from the other terminal via the network 105, decode the coded data, and display the restored video data. Unidirectional data transmission may be common in applications such as media serving.

[0023] Figure 1 shows a second pair of terminals 101 and 104 provided to support the bidirectional transmission of coded video, for example, during a video conference. For bidirectional data transmission, each terminal 101 and 104 may code video data captured at its local location for transmission to the other terminal via the network 105. Each terminal 101 and 104 may also receive coded video data transmitted by the other terminal, decode the coded data, and display the restored video data on a local display device.

[0024] In Figure 1, terminals 101, 102, 103, and 104 may be represented as a server, a personal computer, and a smartphone, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure can be applied to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 represents any number of networks that transmit coded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data over circuit-switched and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network 105 may not be important to the operation of this disclosure unless described below herein.

[0025] Figure 2 shows an example of the application of the disclosed subject matter, illustrating the arrangement of a video encoder and video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other video-enabled applications, including, for example, video conferencing, digital television, and the storage of compressed video on digital media such as CDs, DVDs, and memory sticks.

[0026] The streaming system may include a capture subsystem 203 which can include a video source 201, such as a digital camera, which creates an uncompressed video sample stream 213. The sample stream 213 may be highlighted as having a high data volume when compared to an encoded video bitstream and may be processed by an encoder 202 coupled to the video source 201, which may be a camera as described above. The encoder 202 may include hardware, software, or a combination thereof to enable or implement aspects of the subject of disclosure, as will be described in more detail below. The encoded video bitstream 204 may have a smaller data volume when compared to the sample stream and may be stored in the streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 and retrieve copies 208 and 206 of the encoded video bitstream 204. Client 212 may include a video decoder 211 that decodes a copy 208 of the incoming encoded video bitstream and creates an output video sample stream 210 that can be rendered on a display device 209 or other rendering device (not shown). In some streaming systems, video bitstreams 204, 206, and 208 may be encoded according to specific video coding / compression standards. Examples of these standards are described above and further described herein.

[0027] Figure 3 may be a functional block diagram of a video decoder 300 according to one embodiment of the present disclosure.

[0028] Receiver 302 can receive one or more codec video sequences to be decoded by decoder 300, which in the same or different embodiments may be one coded video sequence at a time, and the decoding of each coded video sequence is independent of other coded video sequences. Coded video sequences may be received from channel 301, which may be a hardware / software link to a storage device that stores encoded video data. Receiver 302 may receive encoded video data together with other data that may be transferred to their respective user entities (not shown), e.g., coded audio data and / or auxiliary data streams. Receiver 302 may isolate coded video sequences from other data. To counteract network jitter, a buffer memory 303 may be coupled between receiver 302 and entropy decoder / parser 304 (hereinafter, "Parser"). Buffer 303 may not be necessary or may be smaller if receiver 302 is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isocyclonas network. For use on best-effort packet networks such as the Internet, a buffer of 303 may be required, can be relatively large, and can be an advantageously adaptive size.

[0029] The video decoder 300 may include a parser 304 for reconstructing symbols 313 from an entropy-coded video sequence. The categories of these symbols may include information used to manage the operation of the decoder 300, and potentially information for controlling rendering devices such as a display device 312, which is not an integral part of the decoder but may be coupled to it. The control information for rendering devices may be in the form of supplemental enhancement information (SEI messages) or video usability information (VUI) parameter set fragments (not shown). The parser 304 can parse / entropy-decode a received coded video sequence. The coding of the coded video sequence may conform to video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, context-dependent or non-context-dependent arithmetic coding, etc. From the coded video sequence, the parser 304 may extract from the coded video sequence a set of at least one subgroup parameters for subgroups of pixels in the video decoder, based on at least one parameter corresponding to a group. Subgroups can include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), and predictive units (PU). The entropy decoder / parser can also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, and motion vectors.

[0030] The parser 304 may perform entropy decoding / analysis operations on the video sequence received from the buffer 303 to create symbols 313. The parser 304 may receive encoded data and selectively decode specific symbols 313. Furthermore, the parser 304 may determine whether a particular symbol 313 should be provided to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra prediction unit 307, or the loop filter 311.

[0031] The reconstruction of symbol 313 may involve multiple different units, depending on the type of coded video picture or its components (interpicture and intrapicture, interblock and intrablock, etc.), as well as other factors. Which units are involved and how can be controlled by parser 304 using subgroup control information parsed from the coded video sequence. The flow of such subgroup control information between parser 304 and the following multiple units is not illustrated for clarity.

[0032] In addition to the functional blocks already mentioned, the decoder 300 may be conceptually subdivided into several functional units, as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with each other and be at least partially integrated. On the other hand, conceptual subdivision into the functional units described below is suitable for illustrating the disclosed subject matter.

[0033] The first unit is the scaler / inverse unit 305. The scaler / inverse unit 305 receives control information from the parser 304 as symbol 313, including the quantized transformation coefficients and the transformation to be used, block size, quantization factor, and quantization scaling matrix. The scaler / inverse unit 305 can output a block containing sample values ​​that can be input to the aggregator 310.

[0034] In some cases, the output samples of the scaler / inverse transform 305 may relate to intracoded blocks, i.e., blocks that do not use predictive information from a previously reconstructed picture but can use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by the intrapicture predictive unit 307. In some cases, the intrapicture predictive unit 307 generates a block of the same size and shape as the block being reconstructed, using already reconstructed surrounding information fetched from the current (partially reconstructed) picture 309. The aggregator 310 may, on a sample-by-sample basis, add the predictive information generated by the intrapredictive unit 307 to the output sample information provided by the scaler / inverse transform unit 305.

[0035] In other cases, the output samples of the scaler / inverse unit 305 may relate to an intercoded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit 306 can access the reference picture memory 308 and fetch samples to be used for prediction. After motion-compensating the fetched samples according to the symbols 313 related to the block, these samples can be added to the output of the scaler / inverse unit by the aggregator 310 to generate output sample information (in this case, called residual samples or residual signals). The address in the reference picture memory format from which the motion-compensated unit fetches prediction samples may be controlled by motion vectors available to the motion-compensated unit, for example, in the form of symbols 313 which may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory when the exact motion vector of a subsample is in use, motion vector prediction mechanisms, etc.

[0036] The output samples of the aggregator 310 can be subjected to various loop filtering techniques in the loop filter unit 311. The video compression technique may include in-loop filtering techniques controlled by parameters contained in the coded video bitstream and made available to the loop filter unit 311 as symbols 313 from the parser 304, but it may also respond to metadata obtained during decoding of earlier parts (in decoding order) of the coded picture or coded video sequence, and may also respond to previously reconstructed and loop-filtered sample values.

[0037] The output of the loop filter unit 311 may be output to the rendering device 312, or it may be a sample stream that can be stored in the reference picture memory 557 for use in future interpicture prediction.

[0038] Some coded pictures, once fully reconstructed, may be used as reference pictures for later prediction. Once a coded picture is fully reconstructed and identified as a reference picture (e.g., by parser 304), the current reference picture 309 can become part of the reference picture buffer 308 and reallocate new current picture memory before starting the reconstruction of subsequent coded pictures.

[0039] The video decoder 300 may perform decoding operations according to a predetermined video compression technique documented in a standard such as ITU-T Rec.H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that it is faithful to the syntax of the video compression technique or standard, as specified in the video compression technique documentation or standard, specifically in the profile documentation within it. The complexity of the coded video sequence may also be required for compliance to be within the range defined by the level of the video compression technique or standard. In some cases, depending on the level, there may be limitations on the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured at 1 million samples per second), maximum reference picture size, etc. The limits imposed by the level may, in some cases, be further limited by the virtual reference decoder (HRD) specification and the metadata for HRD buffer management signaled in the coded video sequence.

[0040] In one embodiment, receiver 302 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by video decoder 300 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, a time layer, a spatial layer, or a signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, or forward error correction codes.

[0041] Figure 4 may be a functional block diagram of a video encoder 400 according to one embodiment of the present disclosure.

[0042] The encoder 400 may receive video samples from a video source 401 (not part of the encoder) that can capture video images coded by the encoder 400.

[0043] The video source 401 may provide a source video sequence coded by an encoder (303) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and a suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source 401 may be a storage device that stores previously prepared video. In a video conferencing system, the video source 401 may be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description will focus on samples.

[0044] According to one embodiment, the encoder 400 can code and compress pictures of a source video sequence in real time or under any other time constraints required by the application to obtain a coded video sequence 410. One function of the controller 402 is to set an appropriate coding speed. The controller controls and is functionally coupled to other functional units, as described below. The coupling is not illustrated for clarity. Parameters set by the controller may include rate control-related parameters (such as picture skip, quantizer, lambda value of rate distortion optimization technique), picture size, group of pictures (GOP) layout, and maximum motion vector search range. Those skilled in the art will be able to easily identify the other functions of the controller 402, as they may relate to the video encoder 400, which is optimized for a particular system design.

[0045] Some video encoders operate in a manner readily recognizable to those skilled in the art as a “coding loop.” In an overly simplified explanation, the coding loop may consist of an encoding portion of an encoder 400 (hereinafter “source coder”) (which is responsible for creating symbols based on the input and reference pictures to be coded) and a (local) decoder 406 built into the encoder 400 which reconstructs the symbols to create sample data, which will also be created by a (remote) decoder (since any compression between symbols and coded video bitstreams is reversible in the video compression techniques considered in the subject of the disclosure). The reconstructed sample stream is input to a reference picture memory 405. Since decoding the symbol stream results in a bit-exact outcome regardless of the decoder’s location (local or remote), the contents of the reference picture buffer are also bit-exact between the local and remote encoders. In other words, the prediction portion of the encoder “sees” the exact same sample values ​​as reference picture samples that the decoder “sees” when using predictions during decoding. This basic principle of the synchronization of a reference picture (and the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is well known to those skilled in the art.

[0046] The operation of the “local” decoder 406 can be the same as that of the “remote” decoder 300, which has already been described in detail above in relation to Figure 3. However, as also briefly referring to Figure 4, since symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy coder 408 and parser 304 can be reversible, the entropy decoding portion of decoder 300, including channel 301, receiver 302, buffer 303, and parser 304, may not be fully implemented in the local decoder 406.

[0047] An observation that can be made at this point is that any decoder techniques present within the decoder, excluding parsing / entropy decoding, must also be present in the corresponding encoder in substantially the same functional form. The description of encoder techniques can be omitted as it is the inverse of the comprehensively described decoder techniques. More detailed explanations are only necessary in specific areas and are provided below.

[0048] As part of its operation, the source coder 403 can perform motion-compensated predictive coding, predictively coding the input frame by referencing one or more previously coded frames from the video sequence, designated as “reference frames”. In this way, the coding engine 407 codes the difference between the pixel blocks of the input frame and the pixel blocks of the reference frame, which may be selected as a predictive reference to the input frame.

[0049] The local video decoder 406 can decode the coded video data of a frame that may be designated as a reference frame, based on symbols created by the source coder 403. The operation of the coding engine 407 may, advantageously, be an irreversible process. If the coded video data can be decoded by a video decoder (not shown in Figure 4), the reconstructed video sequence may be a copy of the source video sequence with some error. The local video decoder 406 can reproduce the decoding process that may be performed by the video decoder on the reference frame and store the reconstructed reference frame in a reference picture memory 405, which may be a cache, for example. In this way, the encoder 400 can locally store a copy of the reconstructed reference frame having common content as the reconstructed reference frame that will be acquired by a far-end video decoder (without transmission errors).

[0050] The predictor 404 may perform a predictive search for the coding engine 407. That is, for a new frame to be coded, the predictor 404 may search the reference picture memory 405 for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., which can function as appropriate predictive references for the new picture. The predictor 404 may operate on a pixel block-by-pixel basis, sample block by sample block, to find appropriate predictive references. In some cases, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory 405, as determined by the search results obtained by the predictor 404.

[0051] The controller 402 can manage the coding operations of the source coder 403, which may be a video coder, and these operations include, for example, setting parameters and subgroup parameters used to encode video data.

[0052] The output of all the aforementioned functional units may be entropy-coded by the entropy coder 408. The entropy coder converts the symbols generated by the various functional units into coded video sequences by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding.

[0053] Transmitter 409 may buffer the coded video sequence created by entropy coder 408 in preparation for transmission over communication channel 411, which may be a hardware / software link to a storage device that will store the encoded video data. Transmitter 409 may merge the coded video data from source coder 403 with other data to be transmitted, such as coded audio data and / or auxiliary data streams (source not shown).

[0054] The controller 402 can manage the operation of the encoder 400. During coding, the controller 402 can assign a specific coded picture type to each coded picture, which can affect the coding technique that may be applied to each picture. For example, a picture may often be assigned as one of the following frame types:

[0055] An intra-picture (I-picture) can be a picture that can be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs enable various types of intra-pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are aware of these variations of I-pictures and their individual uses and characteristics.

[0056] A predictive picture (P-picture) can be a picture that can be coded and decoded using intra-prediction or inter-prediction, which uses up to one motion vector and reference index to predict the sample values ​​for each block.

[0057] A bidirectional predictive picture (B-picture) can be a picture that can be coded and decoded using intra-prediction or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​for each block. Similarly, a multi-predictive picture can use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0058] A source picture can generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block can be coded. Blocks can be coded predictively by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks in picture I can be coded unpredictably, or they can be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks in picture P can be coded unpredictably via spatial prediction or via temporal prediction by referencing one previously coded reference picture. Blocks in picture B can be coded unpredictably via spatial prediction or via temporal prediction by referencing one or two previously coded reference pictures.

[0059] The encoder 400 may be, for example, a video coder, and can perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec.H.265. In its operation, the encoder 400 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the coded video data may conform to the syntax defined by the video coding technique or standard used.

[0060] In one embodiment, the transmitter 409 may transmit additional data along with the encoded video. The source coder 403 may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and redundant slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, and the like.

[0061] FIG. 5 shows a simplified block-style workflow diagram 500 of an exemplary viewport-dependent process in an Omnidirectional Media Application Format (OMAF) that enables 360-degree virtual reality (VR360) streaming described in the Omnidirectional Media Application Format (OMAF).

[0062] In acquisition block 501, if the image data can represent a scene within VR360, video data A such as data of a plurality of images and audio at the same time instance is acquired. In processing block 503, image B at the same time instance i is processed by one or more of stitching, mapping onto an image projected with one or more virtual reality (VR) angles or other angles / viewpoints, and region-wise packing. Further, metadata indicating either such processed information or other information can be created to assist the delivery process and the rendering process.

[0063] Regarding data D, in image encoding block 505, the projected picture is encoded into data E i and configured into a media file. In viewport-independent streaming, in video encoding block 504, the video picture is encoded as data E, for example, a single-layer bitstream v and data B a Regarding data B, the audio data may also be encoded into data E a in audio encoding block 502.

[0064] Data E a 、E v 、and E i All the encoded bitstreams F iand / or F may be stored on a (Content Delivery Network (CDN) / cloud) server, or otherwise may be fully transmitted to the OMAF player 520, typically in a distribution block 507, and fully decoded by a decoder, so that at least a specific area of ​​the decoded picture corresponding to the current viewport is rendered to the user in the display block 516 at angles that the user might be viewing from the head / eye tracking block 508 through the VR image device, with respect to the device's viewport specifications, with respect to various metadata, file playback, and orientation / viewport metadata. A distinct feature of VR360 is that only a viewport may be displayed at any given time, and such a feature can be used to improve the performance of an omnidirectional video system through selective distribution depending on the user's viewport (or any other criterion such as recommended viewport timed metadata). For example, viewport-dependent distribution may be enabled by tile-based video coding, as in an exemplary embodiment.

[0065] As in the encoding block described above, the OMAF player 520 according to an exemplary embodiment processes the data F' and / or F' i Furthermore, with respect to decapsulation of one or more files / segments of the metadata, one or more facets of such encoding are similarly reversed, and in the audio decoding block 510, audio data E' i In the video decoding block 513, the video data E' v , and image data E' in image decoding block 514 i Decode the data B' in audio rendering block 511. a The audio rendering and image rendering block 515 proceeds with the image rendering of data D', and the display block 516 displays data A' in VR360 format according to various metadata such as orientation / viewport metadata. iThe output is generated, and audio data A' is sent to the speaker / headphone block 512. s The following can be output. Various metadata may affect one of the data decoding and rendering processes depending on the various tracks, languages, quality, and views that may be selected by or for the user of the OMAF Player 520, and it should be understood that the order of processing described herein is presented for exemplary embodiments and may be implemented in other orders in other exemplary embodiments.

[0066] Figure 6 shows a simplified block-style content flow process for (coded) point cloud data with viewpoint position and angle-dependent processing (hereinafter referred to as "V-PCC") for capturing / generating / (de)coding / rendering / displaying a 6-degree-of-freedom media. The features described may be used separately or in any order, and among those shown, elements such as encoding and decoding may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits), and it should be understood that one or more processors may execute a program stored on a non-temporary computer-readable medium according to the exemplary embodiment.

[0067] Figure 600 shows an exemplary embodiment for streaming coded point cloud data using V-PCC.

[0068] In volume data acquisition block 601, a real-world visual scene or a computer-generated visual scene (or a combination thereof) may be captured by a set of camera devices or synthesized by a computer as volume data, and the volume data, which may have any format, may be converted to a (quantized) point cloud data format via image processing in the point cloud conversion block 602. For example, data from volume data may be converted to points of points in a point cloud, area by area by pulling one or more of the values ​​described below from the volume data and any associated data into a desired point cloud format according to an exemplary embodiment. According to an exemplary embodiment, the volume data may be, for example, a 3D dataset of 2D images, such as slices from which a 2D projection of the 3D dataset can be projected. According to exemplary embodiments, a point cloud data format may include a representation of data points in one or more different spaces and may be used to represent volume data, and may provide improvements with respect to sampling and data compression, such as with respect to temporal redundancy. For example, point cloud data in x, y, z format may represent color values ​​(e.g., RGB), brightness, intensity, etc., at each of multiple points in the cloud data, and may be used in conjunction with progressive decoding, polygon meshing, direct rendering, and octree 3D representation of 2D quadtree data.

[0069] In the projection block 603 onto the image, the acquired point cloud data may be projected onto a 2D image and encoded as an image / video picture using video-based point cloud coding (V-PCC). The projected point cloud data may consist of attributes, geometry, occupancy maps, and other metadata used to reconstruct the point cloud data using, for example, a painter's algorithm, a raycasting algorithm, a (3D) binary space partitioning algorithm, etc.

[0070] Meanwhile, in the scene generator block 609, the scene generator can generate several metadata to be used to render and display 6-degree-of-freedom (DoF) media, for example, according to the director's intent or the user's preference. Such 6DoF media may include 3D views of the scene, such as 360VR, from rotational changes on the 3D axes X, Y, and Z, in addition to additional dimensions that enable forward, backward, up, and left and right movement within, or at least corresponding to, the point cloud coded data and virtual experience. Scene description metadata defines one or more scenes composed of coded point cloud data and other media data, such as VR360, brightfield, and audio, and may be provided to one or more cloud servers and / or file / segment encapsulation / decapsulation processes, as indicated in Figure 6 and the associated description.

[0071] Following the video encoding block 604 and image encoding block 605, similar to the video and image encoding described above (and, as understood, audio encoding may also be provided as described above), the file / segment encapsulation block 606 processes the coded point cloud data so that it is configured into a media file for file playback, or a sequence of initialization segments and media segments for streaming according to a specific media container file format, such as one or more video container formats, and such a description represents an exemplary embodiment, and among them, for example, may be used with respect to DASH, which will be described later. The file container may also include scene description metadata, such as from the scene generator block 1109, in the file or segment.

[0072] According to an exemplary embodiment, a file is encapsulated according to scene description metadata such that each file includes at least one viewpoint position at one or more time points in 6DoF media and at least one or more angular views at that one or more viewpoint position, so that such a file may be transmitted on request in response to user or author input. Furthermore, according to an exemplary embodiment, a segment of such a file may include one or more parts of such a file, such as a single viewpoint at one or more time points and a portion of 6DoF media indicating the angle at that viewpoint, but these are merely exemplary embodiments and may be modified depending on various conditions such as the network, user, author capabilities and input.

[0073] According to an exemplary embodiment, the point cloud data is divided into multiple independently coded 2D / 3D regions, such as in one or more of the video encoding block 604 and the image encoding block 605. Each independently coded partition of the point cloud data may then be encapsulated in a file / segment encapsulation block 606 as a track within a file and / or segment. According to an exemplary embodiment, each point cloud track and / or metadata track may contain some useful metadata for viewpoint position / angle-dependent processing.

[0074] According to an exemplary embodiment, metadata useful for viewpoint position / angle-dependent processing, such as that contained in a file and / or segment encapsulated with respect to a file / segment encapsulation block, includes one or more of the following: layout information of 2D / 3D partitions having an index; (dynamic) mapping information associating a 3D volume partition with one or more 2D partitions (e.g., any of tiles / tile groups / slice / subpictures); the 3D position of each 3D partition on a 6DoF coordinate system; a representative viewpoint position / angle list; a selected viewpoint position / angle list corresponding to a 3D volume partition; an index of the 2D / 3D partition corresponding to the selected viewpoint position / angle list; quality (rank) information for each 2D / 3D partition; and rendering information for each 2D / 3D partition according to each viewpoint position / angle, for example. Requesting such metadata when requested, such as by a V-PCC player user or instructed by a content creator on behalf of a V-PCC player user, could enable more efficient processing of specific portions of 6DoF media with respect to such metadata, allowing the V-PCC player to deliver higher quality images of the important portions of the media than other portions, rather than delivering unused portions of the 6DoF media.

[0075] From the file / segment encapsulation block 606, a file or one or more segments of a file may be delivered directly to either the V-PCC player 625 or a cloud server such as the cloud server block 607 using a delivery mechanism (e.g., Dynamic Adaptive Streaming over HTTP (DASH)), the cloud server can extract one or more tracks and / or one or more specific 2D / 3D partitions from the file and merge multiple coded point cloud data into a single data.

[0076] According to data such as the position / field of view tracking block 608, if the current field of view position and angle are defined in a 6DoF coordinate system in the client system, then in the cloud server block 607, the field of view position / angle metadata may be delivered from the file / segment encapsulation block 606 or processed in other ways from a file or segment already present on the cloud server. As a result, the cloud server can extract the appropriate partitions from the store file based on metadata from the client system, for example, a V-PCC player 625, and merge them (if necessary). The extracted data can then be delivered to the client as a file or segment.

[0077] With respect to such data, in the file / segment decapsulation block 615, the file decapsulator processes the file or received segment, extracts the coded bitstream, parses the metadata, and in the video decoding and image decoding blocks 610 and 611, the coded point cloud data is then decoded and reconstructed into point cloud data in the point cloud reconstruction block 612, the reconstructed point cloud data can be displayed in the display block 614, and / or, with respect to scene description data, the scene description data may be initially configured in the scene configuration block 613 according to one or more different scene descriptions in accordance with the scene generator block 609.

[0078] Embodiments of this specification can be applied to environments such as two-dimensional or more video conferencing, or environments where hearing aids or karaoke are used, or theater environments where acoustic feedback may occur.

[0079] For example, the ultimate goal of howling suppression is to attenuate the playback signal and transmit only the target signal to the speaker, which in that sense is similar to embodiments relating to acoustic echo cancellation (AEC).

[0080] Given that deep learning is powerful for modeling complex nonlinear relationships and has been successfully introduced to suppress acoustic echoes, embodiments of this specification also employ deep learning to serve as a powerful alternative to address AHS problems, such as the previous inability to treat howling as a kind of noise for audio enhancement, even if deep learning suppresses howling in a streaming and recursive manner.

[0081] According to embodiments of this specification, an aspect that may be called “Deep AHS” is used to address howling suppression. Specifically, AHS can be viewed as a supervised learning problem concerning the overall task of maintaining only the target signal while suppressing playback signals and background noise in microphone recordings. Given that the playback signal and the target signal are highly correlated, embodiments of this specification may use the coupling of the temporal correlation ("corr."), frequency correlation, and channel covariance ("cov.") of the input signals as features to train an attention-based recurrent neural network to estimate a complex ratio filter of the target signal.

[0082] In this disclosure, embodiments consider acoustic feedback suppression (AHS) as a supervised learning problem and provide a deep learning approach called Deep AHS to address it. Deep AHS is trained in a supervised manner that simplifies the problem by transforming the recurrent feedback suppression process into an instantaneous speech separation process and accelerates model training. The methods of the disclosed embodiments utilize a trained attention-based recurrent neural network or its training to extract a target signal from a microphone recording, thus attenuating playback signals that could lead to feedback. For one or more embodiments, different training strategies and a streaming inference method implemented in recurrent mode are utilized to evaluate the performance of the proposed methods for real-time feedback suppression. Deep AHS avoids feedback detection and essentially prohibits the occurrence of feedback, enabling greater flexibility in the design of speech systems. Experimental results demonstrate the effectiveness of the disclosed embodiments for feedback suppression under different scenarios.

[0083] Figure 7 shows an example 700 of a single-channel acoustic amplification system 701 having a microphone and a speaker coupled in the same space 702. The target sound is picked up by the microphone as s(t) and then sent to the speaker for acoustic amplification. The speaker signal x(t) is played back and reaches the microphone as a playback signal indicated as d(t). d(t) = NL(x(t)) * h(t) Equation (1) In this case, NL(.) represents the nonlinear distortion introduced by the speaker, h(t) represents the acoustic path from the speaker to the microphone, and * represents linear convolution.

[0084] Figure 7 also shows the signal flow 703 of an acoustic feedback suppression system according to an embodiment of this specification. For example, if, without any processing, the speaker signal x(t) becomes a delayed and amplified version of y(t), and this playback signal d(t) is repeatedly re-input to the pickup, the corresponding microphone signal at time index t can be expressed as follows: y(t)=s(t)+n(t)+NL[y(t-Δt)·G]*h(t) Equation (2) In this case, n(t) represents background noise, Δt represents the system delay from the microphone to the speaker, and G represents the amplifier gain. The recursive relationship between y(t) and y(t-Δt) causes re-amplification of the playback signal, resulting in an unpleasant, high-pitched feedback loop, known as acoustic howling.

[0085] That being said, feedback doesn't occur instantaneously, but rather recursively. In other words, feedback starts as multiple playback signals, which are amplified to some extent, and then gradually form a high-pitched sound.

[0086] It should be noted that improperly handled acoustic echo (leakage) can also cause feedback, but acoustic feedback is different from acoustic echo. The main difference between acoustic feedback and acoustic echo is that although both are essentially playback signals, feedback is generated gradually, and the playback signal causing the feedback is generated from the same source as the target signal, whereas acoustic echo is usually generated from a different source (far-end speaker), which makes suppressing feedback more difficult.

[0087] Figure 8 shows an exemplary flowchart 800 relating to an embodiment of supervised learning for feedback suppression. Ideally, if the AHS method could always process the microphone recording completely and attenuate all of its playback components before sending it to the speaker, there would be no feedback problem under any circumstances. From the perspective of speech separation, AHS can be viewed as a speech separation problem in which the target signal s(t) is the source to be separated from the microphone signal, similar to how deep learning-based AEC is formulated.

[0088] However, to achieve feedback suppression using deep learning that takes acoustic feedback characteristics into account, the most important issue is that feedback is generated adaptively, and the current input is determined by the previous output. Specifically, as shown in signal flow 703, if there is distortion / leakage in the signal currently being processed, it will affect the playback signal received in the microphone in the next loop d(t+Δt). Ideally, the deep learning model could be trained adaptively by updating parameters at the sample level. However, this requires an enormous amount of computation and is difficult to implement in practical applications.

[0089] Therefore, embodiments of this specification employ Deep AHS to train a model for feedback suppression using supervised learning. Assuming the model is properly trained, the model should attenuate the playback signal in the microphone and transmit only the target voice to the speaker. During model training, embodiments use the actual output in subsequent calculations of the network as shown in signal flow 703.

number

[0090] Next, by using supervised learning, the playback signal d(t) becomes a determined signal that is influenced only by s(t), and the repeated summation of multiple playback signals in equation (2) can be simplified to a single playback. The corresponding microphone signal for model training is written as follows: y(t)=s(t)+n(t)+NL[s(t-Δt)·G]*h(t) Equation (3)

[0091] During teacher-forced learning, the microphone signal is a mixture of the target signal, background noise, and a single determined playback signal. Therefore, the overall problem can be formulated as a speech separation problem. Training Deep AHS using a teacher-forced learning method not only simplifies the overall problem but also reduces the uncertainty introduced into the AHS's adaptive process, resulting in a robust feedback suppression solution.

[0092] According to exemplary embodiments, different training strategies are considered in accordance with the embodiments herein. One example of a simple embodiment is to use the microphone signal of equation (3) directly as input in S801 and set the corresponding s(t) as the training target in S804. Such a training strategy can be employed as a trained model in S806 without using a reference signal ("w / o Ref").

[0093] Another embodiment involves extracting more information from the input in S802 and using that additional extracted information as a reference signal during model training. Thus, the embodiment uses the delayed microphone signal as an additional input in S803, using the amount of delay estimated during the initial stage. Given that the playback signal can be considered as a delayed, scaled, nonlinear version of s(t), using the delayed microphone signal helps the model to better distinguish between the target signal and the playback. Such an embodiment of the training strategy may be referred to as "w Ref".

[0094] Furthermore, the leak

number

number

[0095] Depending on one or more of the embodiments described above, the AHS can be achieved in S808 to varying degrees.

[0096] Details of the network structure are illustrated and explained using Example 1000 in Figure 10 and flowchart 900 in Figure 9. The microphone signal y(t) and reference signal r(t), sampled at 16 kHz in S901, are first divided into 32 ms frames with a 16 ms frame shift in S902. Next, in S903, a 512-point Short-Time Fourier Transform (STFT) is applied to each frame to obtain frequency-domain inputs Y(m, f) and R(m, f) having frame index m and frequency index f, respectively. Then, the normalized log-power spectrum (LPS) is used to obtain the correlation matrix over the time frame and the microphone (log(|Y| 2 ), Φ T _Y, Φ F _Y) and reference signal (log(|R| 2 ), Φ T _R, Φ F Along with the frequency bin of _R), these can be calculated as input features in S904. Here, Φ T _* and Φ F _* is used to capture the temporal and frequency dependence of the signal, helping to distinguish between howling components and tonal components. Input signal Φ C The channel covariances are computed in S905 as another input feature to explain the cross-correlations between them. The concatenation of these features is used in S906 for model training with a linear layer for feature fusion.

[0097] Figure 11 shows a flowchart 1100 relating to the architecture of a Deep AHS for howling suppression according to an embodiment of the present disclosure. For example, as shown in Example 1000, the network consists of three parts, in the first part 1001, a gated recurrent unit (GRU) layer with 257 hidden units and two 1D convolutional layers are used to estimate complex-value filters for playback suppression and playback estimation, respectively, in S1101. The estimates are then applied to the microphone signal Y in S1102.

number

number

[0098] The LPS of these outputs, along with the fused features of the first part, are concatenated at S1103 and fused to function as inputs to the second part 1002. Using another GRU layer and two 1D convolutional layers, input channel Y,

number

number

number

number

[0099] The third part 1003 is for enhancement filter estimation, motivated by the idea of ​​multi-channel signal processing. The embodiment takes input Y and two estimates

number

number

number

number

number

[0100] The loss function for model training is defined as a combination of the scale-invariant signal-to-distortion ratio (SI-SDR) in the time domain and the mean absolute error (MAE) of the spectral amplitude in the frequency domain.

number

[0101] This is because there can always be discrepancies between the offline training and inference phases of Deep AHS. In the streaming inference method, the processor output is looped back and added to the input in the next time step; therefore, the streaming inference method is implemented to evaluate the performance of Deep AHS in a realistic and recursive mode. Details of this streaming inference are shown in Example 1200 in Figure 12.

[0102] Accordingly, embodiments of the present disclosure provide a deep learning method for acoustic feedback suppression. The embodiments address AHS by extracting a target signal from a microphone recording using an attention-based recurrent neural network with appropriately designed features. By the concept of supervised forced learning, the Deep AHS model is trained offline using a training signal and evaluated both offline and streaming to demonstrate feedback suppression performance.

[0103] This disclosure makes four technical contributions. First, Deep AHS formulates howling suppression as an adaptive procedure, using supervised learning with the help of supervised forced learning. This is fundamentally different from conventional AHS methods and does not require howling detection. Second, using such a training strategy, a streaming inference method is implemented to recursively evaluate the performance of Deep AHS. Third, Deep AHS is robust to nonlinear distortion and can achieve howling and noise suppression together under different scenarios, which enables higher loop gain and brings flexibility to the design of speech systems. Finally, multiple training strategies for howling suppression have been studied.

[0104] Embodiments of this disclosure treat acoustic feedback suppression (AHS) as a supervised learning problem and employ a deep learning approach called Deep AHS to address it. Deep AHS is trained in a supervised-forced manner, simplifying the problem by transforming the recurrent feedback suppression process into an instantaneous speech separation process and accelerating model training. Embodiments train an attention-based recurrent neural network to leverage appropriately designed features to extract a target signal from a microphone recording, thus attenuating the playback signal that could cause feedback. Various training strategies are studied, and the performance of the proposed method for real-time feedback suppression is evaluated using a streaming inference method implemented in recurrent mode. Deep AHS avoids feedback detection and essentially prohibits the occurrence of feedback, enabling greater flexibility in the design of speech systems. Experimental results demonstrate the effectiveness of the proposed method for feedback suppression under different scenarios.

[0105] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media, or specifically as computer software configured by one or more hardware processors. For example, Figure 13 shows a computer system 1300 suitable for carrying out a particular embodiment of the disclosed subject matter.

[0106] Computer software can be coded using any suitable machine code or computer language, and may be subject to assembly, compilation, linking, or similar mechanisms, creating code that contains instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), etc., or through interpretation, microcode execution, etc.

[0107] Instructions can be executed on various types of computers or computer components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, and Internet of Things devices.

[0108] The components shown in Figure 13 for computer system 1300 are essentially illustrative and are not intended to imply any limitations on the scope of use or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should not be construed as having any dependencies or requirements relating to any or any combination of components shown in the exemplary embodiment of computer system 1300.

[0109] The computer system 1300 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users via tactile input (such as keystrokes, swipes, or data glove movements), voice input (such as vocalizations or clapping), visual input (such as gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images acquired from still image cameras), or video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0110] The input human interface device may include one or more of the following (but only one of each as shown): keyboard 1301, mouse 1302, trackpad 1303, touchscreen 1310, joystick 1305, microphone 1306, scanner 1308, and camera 1307.

[0111] The computer system 1300 may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., touchscreen 1310, or tactile feedback via joystick 1305, although there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speaker 1309, headphones (not shown)), visual output devices (e.g., screen 1310, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality, some of which can output two-dimensional visual output or three-dimensional hyper-output via means such as stereoscopic output, virtual reality glasses (not shown), holographic display devices, and smoke tanks (not shown)), and printers (not shown).

[0112] The computer system 1300 may also include human-accessible storage devices and their associated media, such as optical media including a CD / DVD ROM / RW 1320 with media like a CD / DVD 1311, a thumb drive 1322, a removable hard drive or solid-state drive 1323, legacy magnetic media such as tape and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), and the like.

[0113] Those skilled in the art will also understand that the term “computer-readable medium” as used in relation to the subject matter protected by this disclosure does not include transmission media, carrier waves, or other transient signals.

[0114] The computer system 1300 may also include an interface 1399 to one or more communication networks 1398. Networks 1398 may be, for example, wireless, wired, or optical. Networks 1398 may further be local, wide-area, metropolitan, automotive and industrial, real-time, latency-tolerant, etc. Examples of networks 1398 include local area networks such as Ethernet, cellular networks including Wi-Fi, GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide-area digital television networks including cable television, satellite television, and terrestrial broadcast television, automotive and industrial networks including CANBus, etc. Certain networks 1398 generally require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (1350 and 1351) (e.g., a USB port on the computer system 1300), and other adapters are generally integrated into the core of the computer system 1300 by connection to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system), as described later. Using any of these networks 1398, the computer system 1300 can communicate with other entities. Such communication can be unidirectional reception only (e.g., broadcast television), unidirectional transmission only (e.g., CANbus to a specific CANbus device), or bidirectional communication to other computer systems using, for example, a local area or wide area digital network. Several protocols and protocol stacks can be used for each of the aforementioned networks and network interfaces.

[0115] The aforementioned human interface device, human-accessible storage device, and network interface may be mounted on the core 1340 of the computer system 1300.

[0116] The core 1340 may include one or more central processing units (CPUs) 1341, graphics processing units (GPUs) 1342, graphics adapters 1317, specialized programmable processing units in the form of field-programmable gate areas (FPGAs) 1343, hardware accelerators 1344 for specific tasks, etc. These devices may be connected via a system bus 1348, along with read-only memory (ROM) 1345, random access memory 1346, internal mass storage such as hard drives (SSDs) 1347 that are not accessible to the user, etc. In some computer systems, the system bus 1348 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 1348 or via a peripheral bus 1349. Architectures for peripheral buses include PCI, USB, etc.

[0117] The CPU 1341, GPU 1342, FPGA 1343, and accelerator 1344 can execute specific instructions that, when combined, constitute the computer code described above. This computer code can be stored in ROM 1345 or RAM 1346. Transitional data can also be stored in RAM 1346, while permanent data can be stored, for example, in internal mass storage 1347. Fast storage and retrieval to any memory device can be enabled by the use of cache memory, which may be closely associated with one or more CPUs 1341, GPUs 1342, mass storage 1347, ROM 1345, RAM 1346, etc.

[0118] Computer-readable media may contain computer code for implementing various computer operations. The media and computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type readily available and well-known to those skilled in computer software technology.

[0119] For example, and not limited to, a computer system having architecture 1300, particularly core 1340, may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with the user-accessible mass storage described above, as well as specific storage of core 1340 of a non-transient nature, such as core internal mass storage 1347 or ROM 1345. Software implementing various embodiments of this disclosure may be stored in such devices and executed by core 1340. The computer-readable media may include one or more memory devices or chips as required by individual needs. The software may cause core 1340, specifically the processor (including a CPU, GPU, FPGA, etc.) within it, to execute specific processes or specific parts of specific processes described herein, which include defining data structures stored in RAM 1346 and modifying such data structures according to processes defined by the software. In addition, or instead, a computer system may provide functionality as a result of logic embodied in a circuit (e.g., accelerator 1344) in a hardwired or otherwise manner, and the circuit may, in place of or in conjunction with the software, perform a particular process or a particular part of a particular process described herein. Where appropriate, “software” may include logic, and vice versa. Where appropriate, “computer-readable medium” may include a circuit (such as an integrated circuit (IC)) that stores the software to be executed, a circuit that implements the logic to be executed, or both. This disclosure includes any appropriate combination of hard and software.

[0120] While this disclosure has described several exemplary embodiments, there are many variations, substitutions, and alternative equivalents that fall within the scope of this disclosure. Those skilled in the art will therefore understand that numerous systems and methods not expressly shown or described herein can be devised to embody the principles of this disclosure and thus fall within the spirit and scope of this disclosure. [Explanation of Symbols]

[0121] 100 Communication system, 101 Terminal, 102 Terminal, 103 Terminal, 104 Terminal, 105 Network, 201 Video source, 202 Encoder, 203 Capture subsystem, 204 Video bitstream, 205 Streaming server, 206 Video bitstream, 207 Streaming client, 208 Video bitstream, 209 Display device, 210 Video sample stream, 211 Video decoder, 212 Streaming client, 213 Uncompressed video sample stream, 300 Decoder, 301 Channel, 302 Receiver, 303 Buffer memory, 304 Parser, 305 Scaler / Inverse unit, 306 Motion compensation prediction unit, 307 Intra prediction unit, 308 Reference picture memory, Reference picture buffer, 309 Picture, 310 Aggregator, 311 Loop filter, 312 Display device, Rendering device, 313 Symbol, 400 Encoder, 401 Source, 402 Controller, 403 Source Coder, 404 Predictor, 405 Reference Picture Memory, 406 Decoder, 407 Coding Engine, 408 Entropy Coder, 409 Transmitter, 410 Video Sequence, 411 Communication Channel, 500 Workflow Diagram, 520 OMAF Player, 557 Reference Picture Memory, 600 Content Flow Process Diagram, 625 V-PCC Player, 700 Example, 701 Single Channel Acoustic Amplification System, 702 Space, 703 Signal Flow, 800 Flowchart, 900 Flowchart, 1000 Example, 1001 Part 1, 1002 Part 2, 1003 Part 3, 1100 Flowchart, 1200 Example, 1300 Computer System, 1301 Keyboard, 1302 Mouse, 1303 Trackpad, 1305 Joystick, 1306 Microphone, 1307 Camera, 1308 Scanner, 1309 Speaker, 1310 Touchscreen, 1311 CD / DVD, 1317 Graphics Adapter, 1320 CD / DVD ROM / RW, 1322 Thumb Drive, 1323 Removable Hard Drive or Solid State Drive, 1340 Core, 1341 Central Processing Unit (CPU), 1342Graphics Processing Unit (GPU), 1343 Field-Programmable Gate Area (FPGA), 1344 Hardware Accelerator, 1345 Read-Only Memory (ROM), 1346 Random Access Memory, 1347 Internal Mass Storage, 1348 System Bus, 1349 Peripheral Bus, 1350 Peripheral Bus, 1351 Peripheral Bus, 1398 Network, 1399 Network Interface

Claims

1. A method for acoustic feedback suppression (AHS), which is performed by at least one processor, The steps include receiving an audio signal from a microphone, The steps include inputting the aforementioned audio signal into a neural network-based AHS model, wherein the neural network-based AHS model is trained using the training audio signal, and inputting the signal. A step of outputting an AHS signal from the neural network-based AHS model to which AHS is applied to the audio signal, wherein the AHS signal is a version of the audio signal in which acoustic feedback noise of the audio signal is suppressed and the target audio of the audio signal is sustained. Methods that include...

2. The neural network-based AHS model is trained with a loss function that includes a combination of the scale-invariant signal-to-distortion ratio (SI-SDR) in the time domain and the mean absolute error (MAE) of the spectral amplitude in the frequency domain. The method according to claim 1.

3. The method according to claim 1, wherein the neural network-based AHS model is trained by supervised learning.

4. The neural network-based AHS model comprises a first gated recurrent unit (GRU) layer configured to apply estimates to the audio signal. The method according to claim 3.

5. The first GRU layer includes 257 hidden units and two one-dimensional (1D) convolutional layers. The method according to claim 4.

6. The neural network-based AHS model further comprises a second GRU layer configured to receive the output of the first GRU and to generate a covariance matrix between the acoustic feedback noise and the target speech. The method according to claim 4.

7. The second GRU layer is configured to receive both the audio signal and the output from the first GRU. The method according to claim 6.

8. The neural network-based AHS model further comprises an enhancement filter estimation layer having a self-attention recurrent neural network (RNN) configured to provide an audio enhancement filter to the input channel of the audio signal. The method according to claim 7.

9. The input to the first GRU layer includes the audio signal, the normalized log-power spectrum (LPS) of the audio signal, the temporal correlation of the audio signal, the frequency correlation of the audio signal, and the channel covariance of the audio signal. The method according to claim 8.

10. The input to the first GRU layer includes the concatenation of the temporal correlation of the audio signal, the frequency correlation of the audio signal, and the channel covariance of the audio signal. The method according to claim 9.

11. A device for video coding, At least one memory configured to store computer program code, At least one processor configured to access the computer program code and to operate as instructed by the computer program code, wherein the computer program code is The at least one processor receives a code configured to receive an audio signal acquired from a microphone, The process involves inputting code configured to cause the audio signal to be input to a neural network-based AHS model, wherein the neural network-based AHS model is trained using the training audio signal. The apparatus includes outputting a code configured to cause at least one processor to output an AHS signal from the neural network-based AHS model to which the AHS is applied to the audio signal, wherein the AHS signal is a version of the audio signal in which acoustic feedback noise of the audio signal is suppressed and the target voice of the audio signal is sustained.

12. The neural network-based AHS model is trained with a loss function that includes a combination of the scale-invariant signal-to-distortion ratio (SI-SDR) in the time domain and the mean absolute error (MAE) of the spectral amplitude in the frequency domain. The apparatus according to claim 11.

13. The apparatus according to claim 11, wherein the neural network-based AHS model is trained by supervised learning.

14. The neural network-based AHS model comprises a first gated recurrent unit (GRU) layer configured to apply estimates to the audio signal. The apparatus according to claim 13.

15. The first GRU layer includes 257 hidden units and two one-dimensional (1D) convolutional layers. The apparatus according to claim 14.

16. The neural network-based AHS model further comprises a second GRU layer configured to receive the output of the first GRU and to generate a covariance matrix between the acoustic feedback noise and the target speech. The apparatus according to claim 14.

17. The second GRU layer is configured to receive both the audio signal and the output from the first GRU. The apparatus according to claim 16.

18. The neural network-based AHS model further comprises an enhancement filter estimation layer, which includes a self-attention recurrent neural network (RNN) configured to provide an audio enhancement filter to the input channel of the audio signal. The apparatus according to claim 17.

19. The input to the first GRU layer includes the audio signal, the normalized log-power spectrum (LPS) of the audio signal, the temporal correlation of the audio signal, the frequency correlation of the audio signal, and the channel covariance of the audio signal. The apparatus according to claim 18.

20. For computers, Receiving audio signals from a microphone, The aforementioned audio signal is input to a neural network-based AHS model, the neural network-based AHS model is trained using the training audio signal, and the input is... The method involves outputting an AHS signal from the neural network-based AHS model to which AHS is applied, wherein the AHS signal is a version of the audio signal in which acoustic feedback noise is suppressed and the target audio of the audio signal is sustained. A non-temporary, computer-readable medium that stores a program to perform a certain action.

Images