Depth audio scaling: beam width controllable neural beamformer

Abstract

Methods and apparatus, the apparatus comprising computer code configured to cause one or more processors to receive a plurality of audio signals, the plurality of audio signals derived from some of a plurality of microphones of a microphone array; performing audio zooming on the plurality of audio signals by selectively focusing and enhancing a plurality of first audio signals of the plurality of audio signals and attenuating a plurality of other audio signals of the plurality of audio signals; and, based on the audio zoom, controlling audio output, wherein the audio zooming comprises combining a plurality of directional characteristics of the plurality of first audio signals of the plurality of audio signals inside a field around the microphone array, and de-suppressing by determining directional characteristics of the plurality of other audio signals of the plurality of audio signals from outside the field.

Description

[0001] This application claims priority to U.S. Application No. 18 / 494,307, filed on October 25, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to depth audio scaling using a neural beamformer with controllable beamwidth. Background Technology

[0003] Audio zoom is a signal processing technique that selectively focuses and enhances sound signals from a specific area while attenuating sound signals from other areas. While traditional beamforming and neural beamforming techniques, which are centered on creating directional arrays, require specifying a single target direction, these techniques may ignore the concept of a field of view (FOV) that defines angular regions.

[0004] In other words, audio zoom technology refers to a signal processing technique that allows users to selectively focus and enhance audio signals originating from specific regions of interest within a sound field, while attenuating signals from other directions. Audio zoom technology has applications in various fields, including teleconferencing, detection, broadcasting, and video recording. In teleconferencing, audio zoom can improve speech intelligibility by isolating sound signals from the target speaker's area and reducing background noise and interference. In detection, audio zoom can be used to detect regions by focusing on sound originating from specific regions of interest, while suppressing sounds from other directions. In broadcasting, audio zoom can enhance the viewer experience by allowing them to selectively listen to narration or sound effects while reducing the volume of other sounds in the broadcast. With audio zoom features, users can use pinch gestures on the screen to adjust the focus of video recordings. When the user zooms in, the audio from the user's focal point becomes more prominent. Conversely, when the user zooms out, ambient sounds become clearer and no longer attenuate.

[0005] Beamforming is the research area most closely related to this concept. It is a computational technique involving the construction of directional microphones using an omnidirectional microphone array. By utilizing the different time delays of signals from different directions, beamforming combines microphone signals into an output signal that amplifies sound from a target direction while suppressing all other sounds, used for monoaural directional sound filtering. In almost all beamforming techniques, it is necessary to specify or estimate a single target direction, which plays a crucial role in the mathematical formula of beamforming. Most beamforming techniques share the same goal of improving sound quality from a single direction. However, these techniques do not consider the concept of field of view (FOV). Furthermore, beamforming typically requires a large microphone array to create a narrow beam that captures sound from a specific direction. The width of the enhanced beam in beamforming is fixed and determined by several factors, including microphone configuration, signal frequency, and the type of beamformer used. An audio zoom implementation has been proposed that combines several robust adaptive beamformers with a derived post-processing algorithm to further enhance the target sound source and weightedly mixes the enhanced target sound source with the original microphone signal to create the audio zoom effect. They also proposed calculating the acoustic spectrum covariance matrix, considering both the desired audio signals inside and outside the FOV, and using these estimated matrices to solve the generalized eigenvalue problem. The resulting beamformer can amplify sound originating from inside the FOV. However, a constraint of these techniques is that in environments with sound reflections, audio waves from sources outside the FOV can bounce off the surface inside the FOV before reaching the microphone. In these cases, their audio zoom technique will continue to amplify these reflected sound signals, which is a technical problem that arises in this technique.

[0006] Therefore, for any of the above reasons, there is an urgent need for technical solutions to these problems in computer audio technology. Summary of the Invention

[0007] According to an embodiment, a method and apparatus are provided. The apparatus includes at least one memory storing program code; at least one processor reading the at least one memory and operating according to the program code, the program code including: receiving code configured to cause the at least one processor to receive a plurality of audio signals, the plurality of audio signals being obtained from some of a plurality of microphones in a microphone array; execution code configured to cause the at least one processor to perform audio zoom on the plurality of audio signals by selectively focusing and amplifying a plurality of first audio signals of the plurality of audio signals and attenuating a plurality of other audio signals of the plurality of audio signals; and control code configured to cause the at least one processor to control audio output based on the audio zoom; wherein the audio zoom includes: incorporating a plurality of directional features of the plurality of first audio signals of the plurality of audio signals within a field surrounding the microphone array, and suppressing the plurality of other audio signals of the plurality of audio signals from outside the field.

[0008] The audio zoom further includes sampling the plurality of audio signals along a predetermined number of directions evenly divided around the microphone array.

[0009] The preset number is 36.

[0010] Merging features from multiple directions is based on determinism:

[0011]

[0012] in, This represents the set of indices for the sectors surrounding the microphone array. The directional features represent some of the multiple directional features, t and f represent the total number of frames and the total number of frequency bands of the complex spectrum of the multiple audio signals, respectively, k represents the preset number, and θ represents the azimuth angle.

[0013] The inhibition is based on the determination that:

[0014]

[0015] in, A second set of indices representing the sectors surrounding the microphone array.

[0016] The audio zoom is achieved by cascading and merging the field of view feature vectors, and the cascading is represented as follows:

[0017]

[0018] in, This represents each of the plurality of directional features.

[0019] The audio zoom is based on post-processing, and is represented as follows:

[0020] .

[0021] The audio zoom is applied to 3D space, wherein... Modified to Where α represents the elevation angle, c represents the speaker in the 3D space, and m represents one microphone of the microphone array.

[0022] The audio zoom is implemented based on a neural network. Attached Figure Description

[0023] Other features, properties, and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, wherein:

[0024] Figure 1 These are schematic diagrams based on embodiments of this application;

[0025] Figure 2 This is a simplified block diagram according to an embodiment of this application;

[0026] Figure 3 This is a simplified schematic diagram according to an embodiment of this application;

[0027] Figure 4 This is a simplified schematic diagram according to an embodiment of this application;

[0028] Figure 5 This is a simplified schematic diagram according to an embodiment of this application;

[0029] Figure 6 This is a simplified schematic diagram according to an embodiment of this application;

[0030] Figure 7 This is a simplified schematic diagram according to an embodiment of this application;

[0031] Figure 8 This is a simplified flowchart according to an embodiment of this application;

[0032] Figure 9 This is a simplified schematic diagram according to an embodiment of this application;

[0033] Figure 10 This is a simplified flowchart according to an embodiment of this application;

[0034] Figure 11 This is a simplified schematic diagram according to an embodiment of this application;

[0035] Figure 12 This is a simplified schematic diagram according to an embodiment of this application. Detailed Implementation

[0036] The features discussed below can be used individually or in any combination in any order. Furthermore, embodiments can be implemented using processing circuitry, such as one or more processors or one or more integrated circuits. In one example, one or more processors execute a program stored on a non-transient computer-readable medium.

[0037] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of this application is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may encode video data locally for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission may be common in media service applications, etc.

[0038] Figure 1 A second pair of terminals 101 and 104 is shown for supporting bidirectional transmission of encoded video, such as during a video conference. For bidirectional data transmission, each terminal 101 and 104 can encode video data captured at a local location for transmission to the other terminal via network 105. Each terminal 101 and 104 can also receive encoded video data sent by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0039] exist Figure 1 In this application, terminals 101, 102, 103, and 104 may be represented as servers, personal computers, and smartphones, but the principles of this application are not limited thereto. Embodiments of this application are applicable to laptops, tablets, media players, and / or dedicated video conferencing equipment. Network 105 refers to any number of networks transmitting encoded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, unless explained below, the architecture and topology of network 105 may be of little importance to the operation of this application.

[0040] As an example, Figure 2 The diagram illustrates the placement of a video encoder and a video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0041] like Figure 2 As shown, the streaming system may include an acquisition subsystem 203, which may include a video source 201, such as a digital camera, for creating an uncompressed sample stream 213. In an embodiment, the sample stream 213 includes samples captured by the digital camera of the video source 201. The sample stream 213 is depicted as a thick line to emphasize a high-data-volume video image stream compared to the encoded video stream, and the sample stream 213 may be encoded by an encoder 202 coupled to the video source 201. The encoder 202 may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video stream 204 is depicted as a thin line to emphasize a lower data volume compared to the sample stream 213, and it may be stored on a streaming server 205 for future use. At least one streaming client 212 and 207 may access the streaming server 205 to retrieve copies 208 and 206 of the encoded video stream 204. Client 212 may include video decoder 212, which decodes the encoded video bitstream 208 and produces an output video sample stream 210 that can be displayed on display 209 or another presentation device (not depicted). In some streaming systems, video bitstreams 204, 206, and 208 may be encoded according to certain video encoding / compression standards. Examples of such standards are described above and further in this application.

[0042] Figure 3 This is a block diagram of a video decoder 300 according to an embodiment disclosed in this application.

[0043] Receiver 302 may receive at least one encoded video sequence to be decoded by video decoder 300. In the same or another embodiment, one encoded video sequence at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequences may be received from channel 301, which may be a hardware / software link to a storage device storing the encoded video data. Receiver 302 may receive encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective user entities (not shown). Receiver 302 may separate the encoded video sequences from other data. To prevent network jitter, buffer memory 303 may be coupled between receiver 302 and entropy decoder / parser 304. Buffer 303 may be unnecessary or can be small when receiver 302 receives data from a store / forward device with sufficient bandwidth and controllability or from an equivalent synchronous network. Buffer 303 may be required for use on best-effort packet networks such as the Internet; buffer 303 may be relatively large and may advantageously have an adaptive size.

[0044] The video decoder 300 may include a parser 304 for reconstructing symbols 313 from the encoded video sequence. These symbols may include information for managing the operation of the video decoder 300, and potential information for controlling display devices such as a display device (312). Control information for the rendering device may be in the form of Supplementary Enhancement Information (SEI) messages or fragments of Video Usability Information (VUI) parameter sets (not shown). The parser 304 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 304 may extract a subset of parameters from the encoded video sequence for at least one subset of pixels in the subgroups used in the video decoder, based on at least one parameter corresponding to a group. Subgroups can include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), and so on. The entropy decoder / parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, and so on.

[0045] The parser 304 can perform entropy decoding / parsing operations on the video sequence received from the buffer 303 to create symbol 313. The parser 304 can receive encoded data and selectively decode specific symbols 313. In addition, the parser 304 can determine whether to provide specific symbols 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra-frame prediction unit 307, or the loop filter 311.

[0046] Depending on the type of encoded video images or their portions (e.g., between and within images, between and within blocks) and other factors, the reconstruction of symbol 313 may involve multiple distinct units. Which units are involved and how they are controlled is determined by subgroup control information parsed from the encoded video sequence by parser 304. For clarity, the flow of this subgroup control information between parser 304 and the multiple units below is not depicted.

[0047] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into multiple functional units as described below. In practical implementations operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated. However, for the sake of describing the disclosed subject matter, it is appropriate to subdivide the concept into the following functional units.

[0048] The first unit is the scaler / inverse transform unit 305. The scaler / inverse transform unit 305 receives quantization transform coefficients as symbols 313 from the parser 304, along with control information including the transform mode used, block size, quantization factor, and quantization scaling matrix. The scaler / inverse transform unit 305 can output blocks containing sample values, which can be input into the aggregator 310.

[0049] In some cases, the output samples of the scaler / inverse transform unit 305 may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 uses reconstructed information extracted from the current (partially re-entrant) image 309 to generate surrounding blocks of the same size and shape as the block being reconstructed. In some cases, the aggregator 310 adds the predictive information generated by the intra-picture prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305 based on each sample.

[0050] In other cases, the output samples of the scaler / inverse transform unit 305 may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit 306 can access the reference image memory 308 to extract samples for prediction. After motion compensation is performed on the extracted samples according to symbol 313, these samples can be added by the aggregator 310 to the output of the scaler / inverse transform unit 305 (referred to as residual samples or residual signals in this case), thereby generating output sample information. The address in the reference image memory form where the motion compensation unit obtains the predicted samples can be controlled by motion vectors, which are provided to the motion compensation unit in the form of symbol 313, which may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0051] The output samples of aggregator 310 can be employed by various loop filtering techniques in loop filter unit 311. Video compression techniques may include in-loop filtering techniques controlled by parameters contained in the encoded video bitstream and provided to loop filter unit 311 as symbols 313 from parser 304, but may also be in response to metadata obtained during decoding of previous (in decoding order) portions of encoded images or encoded video sequences, as well as sample values ​​in response to previous reconstruction and loop filtering.

[0052] The output of the cyclic filter unit 311 can be a sample stream, which can be output to the rendering device 312 or stored in the reference image memory 557 for future inter-image prediction.

[0053] Once a certain encoded image is fully reconstructed, it can be used as a reference image for future predictions. Once an encoded image is fully reconstructed and identified as a reference image (e.g., by parser 304), the current reference image 309 can become part of the reference image buffer 308, and a new current image memory can be reallocated before starting the reconstruction of the next encoded image.

[0054] The video decoder 300 can perform decoding operations according to predetermined video compression techniques that can be documented in standards such as ITU-T Rec.H.265. The encoded video sequence can conform to the syntax specified by the video compression technique or standard used, as it conforms to the syntax of the video compression technique or standard specified in the video compression technique documentation or standard, particularly the configuration file document therein. Compliance also requires that the complexity of the encoded video sequence be within the range defined by the video compression technique or standard level. In some cases, the level limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (e.g., megasamples per second), maximum reference image size, etc. In some cases, the limitations set by the level can be further restricted by the Hypothetical Reference Decoder (HRD) specification and the HRD buffer management metadata emitted in the encoded video sequence.

[0055] In one embodiment, receiver 302 may receive additional (redundant) data with encoded video. The additional data may be included as part of the encoded video sequence. Video decoder 300 may use the additional data to correctly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0056] The embodiments of this application can be applied to environments such as 2D or more-dimensional video conferencing, detection or hearing aid or karaoke environments or theater environments, which may experience a degraded sound quality and can be improved by audio zoom.

[0057] Figure 4 An example 400 of a single-channel sound reinforcement system 401 with a microphone and a speaker coupled in the same space 402 is illustrated. The target speech is picked up by the microphone as... The signal is then sent to the speaker for amplification. It is played back and reaches the microphone as a playback signal, which indicates that... :

[0058] Equation (1)

[0059] in Indicates the nonlinear distortion introduced by the loudspeaker. This represents the sound path from the speaker to the microphone, and Indicates linear convolution.

[0060] Figure 4 The diagram also illustrates the signal flow 403 of the sound quality degradation suppression system according to embodiments of this document. For example, if the speaker signal... It will be The delayed and amplified version, and the playback signal If the microphone is repeatedly re-entered, then in the time index The microphone signal at that location can be represented as:

[0061] Equation (2)

[0062] Where n(t) represents the background noise, Dt indicates the system delay from the microphone to the speaker, and G represents the amplifier gain. and The recursive relationship between them causes the playback signal to be amplified again, resulting in a feedback loop that produces an annoying high-pitched sound, which is known as a form of sound quality degradation.

[0063] Figure 5The illustration shows an example 500 of an application scenario involving a telephone conference according to an exemplary embodiment, wherein, as in example 501, audio zoom can be used to emphasize telephone conference participants at a table, such that sounds originating from the table area are emphasized or amplified, while other sounds (even those reflected from the table area but not originating from it) are attenuated. This can be accomplished through user selection, such as if example 501 represents a detection lens and, according to an exemplary embodiment, the table area is highlighted through user input of the display of that lens, or it can be accomplished automatically, according to an exemplary embodiment, by detecting the majority of speech from the table area. Similarly, as in example 502, alternatively, audio amplification of a person from one side of the table area in example 502 can be performed using a similar technique according to an exemplary embodiment.

[0064] Check Figure 6 In Example 600, the M microphone signals ym input at S601 are transformed into their complex spectrum Ym by a short-time Fourier transform (STFT) at S602, where m = 1, 2, ..., M. At S603, one of the spatial characteristics (IPD, inter-channel phase difference) is calculated as follows using the phase difference between the channels of the complex spectrum:

[0065] Equation (3)

[0066] Where m1 and m2 are the two microphones of the m-th microphone pair out of the M selected microphone pairs.

[0067] At S604, the directional feature (DF) is incorporated as the target speaker bias. According to an embodiment, this feature is used to calculate the mean cosine distance between the target speaker steering vector and the IPD on all selected microphone pairs as follows:

[0068] Equation (4)

[0069] in It is the phase of the target speaker from the θ direction relative to the guide vector of the m-th microphone pair at frequency f. Let be the distance between the m-th microphone pair, c be the speed of sound, and the vector e(·) := [cos(·), sin(·)] T .

[0070] If the TF unit (t, f) is dominated by a source from θ, then It will be close to 1, otherwise it will be biased towards -1. Therefore, Indicating whether the speaker from the desired direction θ is dominant in each TF unit drives the network to extract the target speaker from the mixed signal at S605.

[0071] According to an embodiment, when the space is uniformly divided into several sectors, such as in Figure 7 As shown in the audio zoom example 700 of the 2D FOV feature extraction, the desired angular field of view can cover multiple sectors. The embodiment establishes the concept of "multi-viewpoint" by sampling the direction at the bisector of each sector. The concept of "multi-viewpoint" applies to both speech separation and multi-channel acoustic models, where a small number of spatial viewpoints cover all possible target speaker directions.

[0072] According to the embodiment, neural network-driven multi-view enhancement uses directional features from various view directions as input. This allows the model to simultaneously enhance multiple sampled view directions. A set of K directions in the horizontal plane is sampled. For example, K=36, and the horizontal space is uniformly partitioned into 36 sectors. The azimuth angle θ of the view direction... 1, 2,…,36 ={5°, 15°,…345°, 355°}, corresponding to sectors [0, 10], [10, 20], …, [340, 350] and [350, 360] respectively. The azimuth angle θ of the line of sight. 1, 2,…,K This results in K directional eigenvectors d θk k = 1, 2, ..., K. If the speaker closest to the desired direction dominates in the TF unit, the value of the directional feature in that TF unit is large. Furthermore, this value decreases as the source deviates from the desired line of sight.

[0073] According to an exemplary embodiment, see Figure 8 Example 800 illustrates that for a user-specified FOV depicted by angular boundaries [Θl, Θr], such as when the system receives input to the screen, the embodiment can first determine the corresponding sectors of these boundaries at S801 after receiving the user input. Sectors within the range [Θl, Θr] (inclusive of the boundaries) are defined by their index set. This indicates that, on the other hand, sectors falling outside these boundaries are... This is indicated. Furthermore, this difference can be determined through user input, such as by associating a touch or selection on the screen with a display area on that screen where audio zoom is possible. Regarding directional features... Each of these, where T and F are the total number of frames and the total number of bandwidths in the complex spectrogram, respectively. Through concatenation... The value of , resulting in a considerably large dimension. Furthermore, this dimension fluctuates with changes in field size, making it unsuitable for model training. Therefore, embodiments of this application introduce a method for merging high-dimensional directional feature sets by employing a cross-sector max operation at S802, as detailed in Equation 5:

[0074] Equation (5)

[0075] Simultaneously, at S803, the embodiment incorporates anti-FOV features to represent all directional properties beyond the angular field of view. Using these two features enhances sound capture, particularly near the boundaries of the FOV. This improves the model's ability to distinguish between sound sources within and outside the target field. The anti-FOV features are calculated similarly as follows:

[0076] Equation (6)

[0077] According to the exemplary embodiment, there are two methods to further merge the two FOV feature vectors. At S804, the first method involves concatenation, as shown below:

[0078] Equation (7)

[0079] Furthermore, at S805, the second method involves post-processing, as calculated below:

[0080] Equation (8)

[0081] Post-processing aims to remove speaker components outside the desired region, thereby enhancing the salience of speakers within the region. Figure 9 The audio zoom example shown in Example 900 of 3D FOV feature extraction illustrates 2D feature extraction in the horizontal plane.

[0082] F

[0083] F

[0084] F

[0085] To adapt the above feature extraction to 3D space, the exemplary embodiment modifies the original orientation feature dθ in Equation 4, such that... , where α represents the elevation angle in the vertical dimension. Meanwhile, and The definition will also take into account the preferred elevation angle span. For example... Figure 9 As illustrated, the two lines at angles Θl and Θr delineate the boundaries of the desired angles on the horizontal plane. Meanwhile, angle A... u and A dThe two lines define the boundaries in the vertical plane. The associated sectors are represented by gray in the horizontal and vertical planes, respectively. When considering these angular boundaries in 3D space, the sectors within the field (indicated by...) ) formed by The defined pyramid shape.

[0086] Such as Figure 10 Example 1000's embodiment formulates the problem as amplifying a speech signal captured by a microphone array at S1001 from a specified angular range [Θl, Θr] (using a 2D plane as an example) in an atmosphere of reverberation and background noise. That is, the embodiment uses... This indicates the c-th speaker source received by the m-th microphone. The signal captured by the microphone at time t can be represented as:

[0087] Equation (7)

[0088] This indicates the noise signal received by the m-th microphone. At S1002, the STFT (512 window size and 256 jump size) is calculated using a reference channel (e.g., the complex spectrum Y1 of the first channel) to calculate the logarithmic power spectrum (LPS) at S1003 based on the following formula:

[0089]

[0090] Single-channel spectral characteristics (LPS) and introduced audio zoom FOV characteristics The combination of these elements, at S1004, is merged and used as the input to the neural network.

[0091] The embodiment excludes the DC band, particularly the initial band. Therefore, F is set to 256.

[0092] For each time frame, the input feature vector Ψ is projected onto F via a linear projection layer, from which... Switch to Using the FOV feature outlined in Equation 7 as a reference, according to the embodiment, the total input feature dimension along the frequency axis is equal to 3F.

[0093] After processing by the gated recurrent unit (GRU) encoder layer at S1005, the complex-valued mask of the T×F dimension is inferred at S1006 via four linear layers according to the model of the embodiment: and These are used for source estimation inside and outside the field of view (FOV), respectively. The noise signal is associated with the source outside the FOV, as noise reduction is a crucial component of our objective. At S1007, the two masks are multiplied by the spectrum Ym of each microphone to estimate the target signal and interference signal at each microphone m, yielding... and Then connect in series. and The M channels of the real and imaginary parts are denoted as Φ. According to an embodiment, layer normalization can be used to normalize Φ, which in turn balances the training loss and gradient across blocks within a mini-batch. After a linear projection with leakyRELU activation, the RNN layer, together with the linear layer, receives sub-band embeddings for each frequency band. Where f = 1, 2, ..., F, and E = 32 is the subband embedding dimension, used to estimate the enhancement filter at the corresponding frequency. At S1008, the embodiment is implemented by... Calculate the output signal Finally, the DC band of the output is zero-filled at S1009 to calculate the inverse STFT at S1010, facilitating the generation of the time-domain signal and thus outputting audio with more appropriate audio zoom. The model structure is in Figure 11 The sub-band audio zoom model is shown in Figure 1100.

[0094] Therefore, embodiments of this application represent a simple and effective technique for enabling neural network models to efficiently perform angular region features for audio zoom. Random field-of-view sampling, converting field boundaries to line-of-view, and using a silence-ratio correlation loss function are all key components in achieving neural beamforming with adjustable beamwidth. Experimental results have demonstrated the promise and industrial applicability of the groundbreaking deep learning-based audio zoom approach according to embodiments herein.

[0095] In other words, multichannel technology has gained greater importance as microphone array deployments increase. Microphone arrays can generate multiple recordings carrying spatial information about the sound sources. The effectiveness of established spatial features (such as inter-channel phase difference (IPD)) has been demonstrated for speech separation methods using time-frequency (TF) masks when integrated with monoaural spectral features at the input stage. Additionally, to further improve the extraction of source signals from specific directions, carefully designed directional features indicating the primary directional sources in each time-frequency (TF) unit have been introduced as input features to the neural network model. These directional features depend on the specific direction of interest. Inspired by previous multi-view beamforming and multi-zone beamforming techniques, embodiments of this application introduce FOV features that incorporate all directional features within the desired field of interest. Simultaneously, embodiments integrate inverse FOV features to represent all directional features outside the angular field. Employing both features allows for refinement of sound capture, particularly around the edges of the FOV. In contrast to directional features, FOV features represent the prevalence of all audio sources within a user-specified field of view. When this feature is fed into the neural network, according to an embodiment, the feature ensures that all sound sources or speakers within a specified angular field are captured, while attenuating those sound sources or speakers outside that area.

[0096] Therefore, the key features of the embodiments of this application can be understood as representing the following technical improvements: (1) Field of View (FOV) features (the embodiments of this application introduce a unique FOV feature that merges all directional attributes within a user-defined area or angular field. It encapsulates the sound signal from the specified area, enhancing the capture of the desired sound source.), (2) Inverse FOV features (this is a complementary feature to the FOV feature. The inverse FOV feature captures and represents the directional characteristics outside the desired angular field or area. It helps to correct sound capture, especially around the edges of the FOV.), (3) Integration with a neural network model (FOV and inverse FOV features are designed as input features for a neural network model. This integration ensures effective extraction of sound sources within the specified FOV, emphasizing the capture of all desired sound sources and attenuating unwanted sound sources), and (4) a low-power subband model for real-time applications (the embodiments of this application seamlessly integrate the angular FOV feature into a low-power subband model, and thus apply it to real-time applications according to the embodiments.).

[0097] To achieve these features, the exemplary embodiments provide the following improved functions: (1) selective sound enhancement (the embodiments of this application allow users to selectively focus and enhance audio signals from a specific area while attenuating signals from undesired directions), (2) isolation of target sounds (particularly useful in applications such as teleconferences, this feature can isolate sounds from a target speaker, thereby improving speech clarity and reducing background noise), (3) sound detection in a specific area (in a detection scenario, the audio zoom feature focuses on sounds originating from a specific area of ​​interest, suppressing unwanted noise from other areas), (4) enhanced audiovisual experience (in broadcasting, viewers can selectively listen to specific sound effects or narration, enhancing their overall experience), and (5) adaptive sound capture (the embodiments of this application adapt based on user preferences, ensuring that sounds from desired areas are clear and that ambient sounds are no longer attenuated when the user zooms out).

[0098] The exemplary embodiments described herein are applied to various scenarios, such as (1) telephone conferences (which improve the intelligibility of speech by isolating the audio signal from the area of ​​the target speaker and reducing background noise). Figure 5 The illustration shows sound capture within one of several user-specified areas indicated by the pyramid shape shown. (2) Detection (focusing on a specific region of interest, capturing sound from those areas and attenuating sound from irrelevant areas.), (3) Broadcasting (providing an enhanced audience experience by allowing selective listening to specific audio clips or narration.), (4) Video capture (users can use pinch gestures on the screen to adjust the focus of video recording and correspondingly zoom in or out of the corresponding audio.), (5) Speech separation method (combining the FOV spatial features of the invention, the invention can be used for speech separation methods utilizing time-frequency (TF) masks.), (6) Directional sound filtering (particularly in environments with sound reflections, the invention helps to filter and amplify the desired sound source even when reflected sound signals are present.).

[0099] Therefore, embodiments of this application provide simple yet effective FOV features that incorporate all directional attributes within a user-defined field. Simultaneously, embodiments have introduced anti-FOV features to capture directional aspects outside the desired field. These advancements ensure corrected sound capture, particularly emphasizing the boundaries of the FOV, and guarantee enhanced capture of all desired sound sources within the user-defined field. Experimental results have demonstrated the effectiveness of the introduced angular FOV features and their seamless integration into low-power subband models suitable for real-time applications.

[0100] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media, or implemented by one or more specially configured hardware processors. For example, Figure 12 A computer system 1200 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0101] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored in at least one computer-readable storage medium. For example, Figure 12 A computer system 1200 is shown, which is adapted to implement certain embodiments of the disclosed subject matter.

[0102] The computer software can be encoded using any suitable machine code or computer language, and code including instructions can be created through mechanisms such as assembly, compilation, and linking. These instructions can be executed directly by at least one computer central processing unit (CPU), graphics processing unit (GPU), or through decoding, microcode, or other means.

[0103] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0104] Figure 12 The components shown for computer system 1200 are exemplary in nature and are not intended to limit the scope or functionality of the computer software used to implement the embodiments of this application. Nor should the configuration of the components be construed as having any dependency or requirement on any component or combination thereof shown in the exemplary embodiments of computer system 1200.

[0105] Computer system 1200 may include certain human-computer interface (HCI) input devices. These HCI input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The HCI device may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0106] Human-machine interface input devices may include one or more of the following (only one of each is shown): keyboard 1201, mouse 1202, touchpad 1203, touch screen 1210, joystick 1205, microphone 1206, scanner 1208, and camera 1207.

[0107] The computer system 1200 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. These HMI output devices may include tactile output devices (e.g., tactile feedback from a touchscreen 1210 or joystick 1205, but may also include tactile feedback devices that are not input devices), audio output devices (e.g., speakers 1209, headphones (not shown)), and visual output devices (including screens 1210 such as CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability, each with or without tactile feedback capability, some of which may output two-dimensional or higher visual outputs via stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke canisters (not shown), as well as printer drawing).

[0108] The computer system 1200 may also include human-accessible storage devices and related media, such as optical media, including CD / DVD ROM / RW 1220 or similar media with CD / DVD 1211, thumb drives 1222, removable hard disk drives or solid-state drives 1223, conventional magnetic media such as magnetic tapes and floppy disks (not shown), devices based on dedicated ROM / ASIC / PLD, such as security dongles (not shown), etc.

[0109] Those skilled in the art should also understand that the term "computer-readable storage medium" as used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0110] Computer system 1200 may also include an interface 1299 to one or more communication networks 1298. Network 1298 may be, for example, wireless, wired, or optical. Network 1298 may also be local, wide area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of network 1298 include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television, and vehicle and industrial networks including CANbus, etc. Some networks 1298 typically require an external network interface adapter (e.g., a USB port on computer system 1200) to connect to certain unified data ports or peripheral buses (1250 and 1251); other networks are typically integrated into computer system 1200 via a connection to a system bus (e.g., an Ethernet interface on a PC computer system or a cellular network interface on a smartphone computer system). Using any of these networks 1298, computer system 1200 can communicate with other entities. This communication can be unidirectional, receive-only (e.g., broadcast television), transmit-only (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as with other computer systems using local area networks or wide area digital networks. As mentioned above, certain protocols and protocol stacks are available for each of these networks and network interfaces.

[0111] The aforementioned human-machine interface device, human-accessible storage device, and network interface can be connected to the core 1240 of the computer system 1200.

[0112] Core 1240 may include one or more central processing units (CPUs) 1241, graphics processing units (GPUs) 1242, graphics adapters 1212, dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) 1243, hardware accelerators 1244 for certain tasks, etc. These devices, along with read-only memory (ROM) 1245, random access memory 1246, and internal mass storage (such as internal non-user-accessible hard disk drives, SSDs, etc.) 1247, can be connected via system bus 1248. In some computer systems, system bus 1248 may be accessed as one or more physical connectors to allow for expansion via additional CPUs, GPUs, etc. Peripheral devices can be directly connected to the core's system bus 1248 or connected via peripheral bus 1249. Peripheral bus architectures include PCI, USB, etc.

[0113] The CPU 1241, GPU 1242, FPGA 1243, and accelerator 1244 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 1245 or RAM 1246. Transient data can also be stored in RAM 1246, while permanent data can be stored, for example, in internal mass storage 1247. By using a cache memory, fast storage and retrieval of any storage device can be achieved. The cache memory can be closely associated with one or more CPUs 1241, GPU 1242, mass storage 1247, ROM 1245, RAM 1246, etc.

[0114] The computer-readable storage medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be specifically designed and constructed for the purposes of this application, or they may be media and code well-known and usable by those skilled in the art of computer software.

[0115] By way of example and not limitation, a computer system having architecture 1200, particularly core 1240, can provide functionality by executing software contained in one or more tangible computer-readable media via a processor (including CPU, GPU, FPGA, accelerator, etc.). Such a computer-readable medium may be a medium associated with a user-accessible mass storage as described above, and some memory of core 1240 having non-transient characteristics, such as core-internal mass storage 1247 or ROM 1245. Software implementing various embodiments of this application may be stored in such a device and executed by core 1240. Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software may cause core 1240, particularly its processor (including CPU, GPU, FPGA, etc.), to execute a specific process or a specific portion of a specific process described in this application, including defining data structures stored in RAM 1246 and modifying these data structures according to a software-defined process. In addition, or alternatively, the computer system may provide functionality as a result of hard-wired or otherwise embodied logic in a circuit (e.g., accelerator 1244) that may replace or operate with software to perform a particular process or a particular portion of a particular process described in this application. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry (such as an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both. This application includes any suitable combination of hardware and software.

[0116] While this application has described several exemplary embodiments, various modifications, arrangements, and equivalent substitutions of the embodiments are all within the scope of this application. Therefore, it should be understood that those skilled in the art can design various systems and methods that, although not explicitly shown or described in this application, embody the principles of this application and are thus within its spirit and scope.

Claims

1. An audio processing method, characterized in that, The method, executed by at least one processor, includes: Receive multiple audio signals, which are obtained from some of the multiple microphones in a microphone array; Audio zoom is performed on the plurality of audio signals by selectively focusing and enhancing a plurality of first audio signals and attenuating a plurality of other audio signals; and, Based on the aforementioned audio zoom, control the audio output; The audio zoom includes: incorporating multiple directional features of multiple first audio signals of multiple audio signals within a field surrounding the microphone array, and suppressing them by determining the directional features of multiple other audio signals of multiple audio signals from outside the field.

2. The method according to claim 1, characterized in that, The audio zoom further includes sampling the plurality of audio signals along a predetermined number of directions evenly divided around the microphone array.

3. The method according to claim 2, characterized in that, The preset number is 36.

4. The method according to claim 2, characterized in that, Merging features from multiple directions is based on determinism: ， in, This represents the set of indices for the sectors surrounding the microphone array. The directional features represent some of the multiple directional features, t and f represent the total number of frames and the total number of frequency bands of the complex spectrum of the multiple audio signals, respectively, k represents the preset number, and θ represents the azimuth angle.

5. The method according to claim 4, characterized in that, The inhibition is based on the determination that: ， in, A second set of indices representing the sectors surrounding the microphone array.

6. The method according to claim 5, characterized in that, The audio zoom is achieved by cascading and merging the field of view feature vectors, and the cascading is represented as follows: ， in, This represents each of the plurality of directional features.

7. The method according to claim 5, characterized in that, The audio zoom is based on post-processing, and is represented as follows: 。 8. The method according to claim 4, characterized in that, The audio zoom is applied to 3D space, wherein... Modified to Where α represents the elevation angle, c represents the speaker in the 3D space, and m represents one microphone of the microphone array.

9. The method according to claim 1, characterized in that, The audio zoom is implemented based on a neural network.

10. The method according to claim 1, characterized in that, Audio is output based on the audio zoom during the conference call.

11. An audio processing apparatus, characterized in that, include: At least one memory is used to store program code; At least one processor reads the at least one memory and operates according to the program code, the program code including: The receiving code is configured to cause the at least one processor to receive a plurality of audio signals, the plurality of audio signals being obtained from some of the microphones in a microphone array; The executable code is configured to cause the at least one processor to perform audio zoom on the plurality of audio signals by selectively focusing and amplifying a plurality of first audio signals and attenuating a plurality of other audio signals; and, Control code is configured to cause the at least one processor to control audio output based on the audio zoom; The audio zoom includes: incorporating multiple directional features of multiple first audio signals of multiple audio signals within a field surrounding the microphone array, and suppressing them by determining the directional features of multiple other audio signals of multiple audio signals from outside the field.

12. The apparatus according to claim 11, characterized in that, The audio zoom further includes sampling the plurality of audio signals along a predetermined number of directions evenly divided around the microphone array.

13. The apparatus according to claim 12, characterized in that, The preset number is 36.

14. The apparatus according to claim 12, characterized in that, Merging features from multiple directions is based on determinism: ， in, This represents the set of indices for the sectors surrounding the microphone array. The directional features represent some of the multiple directional features, t and f represent the total number of frames and the total number of frequency bands of the complex spectrum of the multiple audio signals, respectively, k represents the preset number, and θ represents the azimuth angle.

15. The apparatus according to claim 14, characterized in that, The inhibition is based on the determination that: ， in, A second set of indices representing the sectors surrounding the microphone array.

16. The apparatus according to claim 15, characterized in that, The audio zoom is achieved by cascading and merging the field of view feature vectors, and the cascading is represented as follows: ， in, This represents each of the plurality of directional features.

17. The apparatus according to claim 15, characterized in that, The audio zoom is based on post-processing, and is represented as follows: 。 18. The apparatus according to claim 14, characterized in that, The audio zoom is applied to 3D space, wherein... Modified to Where α represents the elevation angle, c represents the speaker in the 3D space, and m represents one microphone of the microphone array.

19. The apparatus according to claim 11, characterized in that, The audio zoom is implemented based on a neural network.

20. A non-volatile computer-readable medium, characterized in that, The computer stores a program that enables the computer to: Receive multiple audio signals, which are obtained from some of the multiple microphones in a microphone array; Audio zoom is performed on the plurality of audio signals by selectively focusing and enhancing a plurality of first audio signals and attenuating a plurality of other audio signals. and, Based on the aforementioned audio zoom, control the audio output; The audio zoom includes: incorporating multiple directional features of multiple first audio signals of multiple audio signals within a field surrounding the microphone array, and suppressing them by determining the directional features of multiple other audio signals of multiple audio signals from outside the field.

Images