System and method for sound reproduction

Abstract

A sound reproduction method includes: receiving an input audio signal representing audio content to be reproduced; generating at least two sound regions using a plurality of speakers, the plurality of speakers being configured, positioned, and operated to spatially limit the audibility of the audio content to be reproduced to one of the at least two sound regions; and reproducing the audio content to be reproduced in the one of the at least two sound regions using at least some of the plurality of speakers, wherein the at least some of the plurality of speakers are configured, positioned, and operated to generate sound from the input audio signal that produces a three-dimensional audio effect, the three-dimensional audio effect including placing a virtual sound source at any location in a three-dimensional space, and the three-dimensional space being one of the at least two sound regions.

Description

Technical Field

[0001] This disclosure relates to a system and method for sound reproduction (collectively, the “System”). Background Technology

[0002] 3D audio rendering technology plays a crucial role in delivering spatial and immersive audio. It is well-known that technology can present users with realistic and immersive sound. However, a significant challenge of 3D audio rendering in listening rooms such as vehicle interiors (e.g., cars) is that the optimal listening experience is often limited to certain locations or "best seats." For example, 3D audio rendering commonly found in car interiors typically provides the best audio experience for passengers in the front seats or specially designated "VIP" seats. Overcoming this limitation is desired. Summary of the Invention

[0003] A sound reproduction method and system includes the following operations: receiving one or more input audio signals representing audio content to be reproduced; generating at least two sound regions using a plurality of speakers, the plurality of speakers being configured, positioned, and operated to spatially limit the audibility of the audio content to be reproduced to at least one of the at least two sound regions; and reproducing the audio content to be reproduced in the at least one of the at least two sound regions using at least some of the plurality of speakers, wherein the at least some of the plurality of speakers are configured, positioned, and operated to generate sound from the input audio signals that produces a three-dimensional audio effect, the three-dimensional audio effect including placing a virtual sound source at any location in a three-dimensional space, and the three-dimensional space being one of the at least two sound regions.

[0004] Other systems, methods, features, and advantages will be or will become apparent to those skilled in the art upon review of the following detailed description and accompanying drawings. It is intended that all such other systems, methods, features, and advantages be included within this specification, within the scope of the invention, and protected by the appended claims. Attached Figure Description

[0005] The system can be better understood by referring to the following figures and description. The components in the figures are not necessarily to scale, but rather to emphasize the principles of the invention. Furthermore, in the figures, the same reference numerals refer to corresponding parts in different views.

[0006] Figure 1 This is a schematic diagram illustrating an example audio system that includes an audio processing system and a speaker.

[0007] Figure 2 This shows what can be applied to Figure 1The diagram shows a block diagram of an example processing structure for an audio reproduction method in an audio processing system.

[0008] Figure 3 This is a top view of a carriage with two separate sound zones.

[0009] Figure 4 This is a schematic diagram showing a 2×2 transaural stereo system.

[0010] Figure 5 It is a top view of the vehicle interior, showing multiple speakers arranged in each individual sound zone around the user's head position.

[0011] Figure 6 This is a schematic diagram illustrating an example of a 3D audio rendering method using upmixing, including translation estimation, direct / ambience decomposition, and re-translation.

[0012] Figure 7 This is a three-dimensional view showing the speaker setup of a nine-channel 3D audio reproduction system.

[0013] Figure 8 This is a schematic diagram illustrating a five-channel audio system using a re-translated virtual source distribution.

[0014] Figure 9 This is a schematic diagram illustrating an example source extraction method using spatial extraction.

[0015] Figure 10 This is a block diagram illustrating the process of generating ambient sound in a virtual space within a listening room.

[0016] Figure 11 It shows the use of Figure 10 The flowchart shown illustrates the process structure for generating a virtual space within a listening room.

[0017] Figure 12 This is a block diagram illustrating an example user preference handling process using a graphical interface and machine learning.

[0018] Figure 13 It is shown Figure 2 The flowchart shows an example workflow of the process structure. Detailed Implementation

[0019] Figure 1This is an example audio system 101 that includes an audio processing system 102. The audio system 101 may also include at least one audio content source 103, a multi-channel amplifier 104, and multiple speakers 105. The audio system 101 can be any system capable of producing audible audio content. Example audio systems 101 include vehicle audio systems, fixed consumer audio systems (such as home theater systems), audio systems for multimedia systems (such as cinemas or televisions), multi-room audio systems, public address systems (such as those in stadiums or conference centers), outdoor audio systems, or any other location where audible audio sound is desired to be reproduced.

[0020] Audio content source 103 can be any form of one or more devices capable of generating and outputting different audio signals on one or more channels. Examples of audio content source 103 include media players (such as optical disc or video disc players), video systems, radios, cassette players, wireless or wired communication devices, navigation systems, personal computers, codecs (such as MP3 players), or any other form of audio-related device capable of outputting audio signals.

[0021] exist Figure 1 In this configuration, audio content source 103 generates two or more audio signals from source material, such as pre-recorded audible sound, on corresponding audio input channels 106. The audio signals can be audio input signals generated by audio content source 103, and can be analog signals based on analog source material or digital signals based on digital source material. Therefore, audio content source 103 can include signal conversion functions, such as analog-to-digital converters or digital-to-analog converters. In one example, audio content source 103 can generate a stereo audio signal consisting of two substantially different audio signals provided as a right and left channel on two audio input channels 110. In another example, audio content source 103 can generate more than two audio signals on more than two audio input channels 106, such as 5.1 surround sound, 6.1 surround sound, 7.1 surround sound, or any other number of different audio signals generated on the same number of corresponding audio input channels 106.

[0022] Amplifier 104 can be any circuit or standalone device that receives a relatively small-amplitude audio input signal and outputs a similar audio signal with a relatively large amplitude. Two or more audio input signals can be received on two or more amplifier input channels 107 and output on two or more audio output channels 108. In addition to amplifying the amplitude of the audio signals, amplifier 104 may also include signal processing capabilities to phase-shift, adjust frequency equalization, adjust delay, or perform any other form of manipulation or adjustment of the audio signals. Furthermore, amplifier 104 may also include the ability to adjust the volume, balance, and / or fade-in / fade-out of the audio signals provided on the audio output channels 108. In alternative examples, the amplifier may be omitted when the audio output channel is used as an input to another audio device. In still some examples, speaker 105 may include an amplifier, such as when speaker 105 is a self-powered speaker.

[0023] The speaker 105 can be positioned in a listening space, such as a room, a vehicle, or any other space where the speaker 105 can be operated. The speaker 105 can be of any size and can operate within any frequency range. Each audio output channel 108 can supply a signal to drive one or more speakers 105. Each of the speakers 105 may include a single transducer or multiple transducers. The speaker 105 can also operate in different frequency ranges, such as a subwoofer, woofer, midrange speaker, and tweeter. The audio system 101 may include two or more speakers 105.

[0024] The audio processing system 102 can receive an audio input signal from the audio content source 103 on the audio input channel 106. After processing, the audio processing system 102 provides the processed audio signal on the amplifier input channel 107. The audio processing system 102 can be a standalone unit or can be combined with the audio content source 103, the amplifier 104, and / or the speaker 105. Furthermore, in other examples, the audio processing system 102 can communicate via a network or communication bus to interface with the audio content source 103, the audio amplifier 104, the speaker 105, and / or any other device or mechanism (including other audio processing systems 102).

[0025] The audio processing system 102 may include one or more audio processors 109. The one or more audio processors 109 may be one or more computing devices capable of processing audio and / or video signals, such as a computer processor, microprocessor, digital signal processor, or any other device, series of devices, or other mechanism capable of performing logical operations. The one or more audio processors 109 may operate in association with memory 110 to execute instructions stored in memory 110. Instructions may be in the form of software, firmware, computer code, or some combination thereof, and when executed by the one or more audio processors 109, provide the functionality of the audio processing system 102. Memory 110 may be one or more data storage devices of any form, such as volatile memory, non-volatile memory, electronic memory, magnetic memory, optical memory, or any other form of data storage device. In addition to instructions, operating parameters and data may also be stored in memory 110. The audio processing system 102 may also include electronic, electromechanical, or mechanical devices, such as devices for converting between analog and digital signals, filters, user interfaces, communication ports, and / or any other functions operating within the audio system 101 and accessible to users and / or programmers.

[0026] During operation, the audio processing system 102 receives and processes signals on the audio input channel 106. Typically, during the processing of signals on the audio input channel 106, one or more audio processors 109 identify multiple perceptual locations for each of a plurality of audible sound sources represented within the audio input signal. These perceptual locations represent the physical location of the corresponding audible sound source within the user's perceived sound field. Thus, if the user (audience) is watching a live performance on an actual stage, the perceptual location will correspond to the position of the performer on stage, such as a guitarist, drummer, singer, and any other performer or object that produces sound in the audio signal.

[0027] The audio processor 109 decomposes the audio input signal into a set of spatial audio streams or spatial slices, each slice containing audio content from a corresponding (at least one) perceptual location within the perceived location. Any sound source located at the same location within a given perceptual location can be included in the same spatial audio stream. Any number of different spatial audio streams can be created within the user-perceived sound field. The spatial audio streams can be processed independently by the audio processor 109.

[0028] During operation, audio processor 109 can generate multiple filters for each of a plurality of corresponding output channels based on the perceived location of the identified corresponding audible sound source. Audio processor 109 can apply the filters to the audio input signal to generate a spatial audio stream. The spatial audio stream can be processed independently. After processing, the spatial audio stream can be assembled or otherwise recombinated to generate an audio output signal with multiple corresponding audio output channels. The audio output channels are located on the amplifier input lines, i.e., audio output channel 108. Audio processing system 102 can provide more or fewer audio output channels than the number of input channels included in the audio input signal. Alternatively, audio processing system 102 can provide the same number of audio output channels as provided as the input channels.

[0029] Furthermore, the audio processor 109 can process the audio signal to be output to the amplifier input channel 107, so that the audibility of different audio content when reproduced via the speaker 105 is spatially confined to different sound zones. For example, multiple users in the same room can be supplied with different audio content through the speaker 105 without the need for headphones or structural acoustic barriers.

[0030] Figure 2 It shows having Figure 1 The audio processing system 102 shown illustrates an example processing structure of processing blocks and their interconnections, designed to allow each user in a group of users at a given location (such as a room) to experience three-dimensional (3D) audio rendering in their unique style. The example process structure involves generating four sound regions 201-204, analogous to an audio rendering scene inside a vehicle with four seating positions. For simplicity, all four regions 201-204 are identical except for their location, but they can be different if desired. In the following description, only region 201 is described in detail as representative of all regions 201-204. The processing structure includes a sound region control block 205, which... Figure 2 The block 201 is shown as common to all sound regions 201-204, but may also exist in each of the sound regions 201-204. Sound region 201 (and each of the other sound regions 202-204) includes a 3D rendering block 206 and a user preference evaluation block 207. As used herein, the term "block" or "multiple blocks" is defined as software (computer code, instructions) or hardware (such as circuits, electronic components, and / or logic), or a combination of software and hardware.

[0031] The sound zone control block 205 is designed to use multiple speakers within the vehicle, combined with specific signal processing (including filtering and delaying the signal to be reproduced), to establish and control sound zone 201 (as well as other sound zones 202-204). The sound zone control block 205 not only uses active noise cancellation technology but also sound guidance technology to focus sound into designated areas while simultaneously minimizing sound energy in other areas. This can also be achieved using multiple microphones (…). Figure 2 (Not shown in the image) is used for sound control. For individual sound zones within a listening environment, each zone reproduces a different sound. To achieve individual sound zones, the responses of multiple sound sources need to be adjusted to approximate the desired sound field in the reproduced zone without interfering with other independent sound zones. An example of how to achieve individual sound zones is provided below. Figure 3 and Figure 4 The description is provided and detailed in U.S. Patent No. 9,338,554, which has been assigned to the assignee of this disclosure and is incorporated herein by reference.

[0032] For example, 3D rendering block 206 generates an output multi-channel 3D audio signal from the input stereo signal (which is two-channel audio) through source extraction block 208, ambient sound generation block 209, and source allocation block 210. Blocks 208-210 can be integrated into the upmixing process. Figure 2 (Not shown in the image). Source extraction block 208 is designed to extract the sources (original sources) present in the input stereo signal, for example, using techniques such as center extraction, vocal extraction, and instrument extraction to extract the center and residual signals from the input stereo signal. Ambient sound generation block 209 provides various ambient sound settings that can be controlled by the user via user preference processing block 207, and includes the generation of, for example, early reflections and reverberation, delay, desired sound pressure level patterns, and pitch. Source assignment block 210 is used to locate the detected sources in the output 3D audio signal and is designed to derive translation coefficients based on user preferences to virtually locate the extracted original sources in the sound to be reproduced. Ambient sound refers to the (environmental) sound of a given location or space. It is the opposite of "silence." Ambient sound is similar to presence, but the difference is that ambient sound contains definite background noise.

[0033] User preference processing block 207 executes, for example, a machine learning (ML)-based algorithm that maps input user preferences related to the virtual source distribution to corresponding parameters of 3D rendering block 206. The machine learning-based algorithm can be trained using various audio signals and a large number of users. User preference evaluation block 208 can perform at least one of the following: retrieving source distribution-related information specified by the user, mapping user information to appropriate audio signal parameters using a machine learning algorithm, and feeding information (e.g., regarding the location of each virtual source desired by the user) to 3D rendering block 206. Sound region control block 205, 3D rendering block 206, and user preference evaluation block 208 can be implemented in various ways, some of which will be described in detail below.

[0034] pass Figure 3 The illustrated example sound zone setup presents the concept of two (or more) separate sound zones in a listening room (e.g., inside a vehicle). Two distinct sound zones (i.e., first sound zone 301 and second sound zone 302) are arranged at two different locations within the listening room 303. In the first sound zone 301, a first audio signal (e.g., speech) is reproduced, while in sound zone 302, a second audio signal (e.g., music) is reproduced. The spatial locations of the two zones 301 and 302 can be fixed or adaptively changed (e.g., depending on the location of the respective user). The objective is to minimize crosstalk from the first audio signal (associated with the first sound zone 301) to the second sound zone 302 and from the second audio signal (associated with the second sound zone 302) to the first sound zone 301. It should be noted that... Figure 3 A top view is shown. Sound areas 301 and 302 actually encompass a three-dimensional volume including the user's head (particularly the ears).

[0035] Figure 4 The basic signal flow structure of a so-called "transear stereo" arrangement is shown. In the depicted system, the audio signal and transfer function are frequency domain signals and functions, respectively, and they have corresponding time domain signals and functions. Left input audio signal X L (jω) and the right input audio signal X R (jω) (which may be provided by, for example, a radio receiver) is supplied by the so-called inverse filter C. LL (jω), C LR (jω), C RL (jω) and C RR (jω) is used for pre-filtering, and the filter output signal is as follows: Figure 2 The combination is shown; that is, the supply is made to the left speaker LS. L signal S L (jω) can be calculated as:

[0036]

[0037] And supplied to the right speaker LS R signal S R (jω) can be calculated as follows:

[0038]

[0039] Speaker radiated signal S L (jω) and S R (jω) serves as the acoustic signal, which propagates to the user's left and right ears, respectively. The actual sound signals perceived by the user's left and right ears are represented as follows: ZL (jω) and ZR (jω), where:

[0040]

[0041] and

[0042]

[0043] In Equations 3 and 4, the transfer function Hij(jω) represents the room impulse response (RIR) in the frequency domain, i.e., from the loudspeaker LS. i and LS j Each of these is a transfer function to the user's left and right ears. Indices i and j can each be "L" or "R", where "L" and "R" refer to the left and right speakers and the ear, respectively. Equations 1-4 above can be rewritten in matrix form, where equations 1 and 2 can be combined into:

[0044]

[0045] Furthermore, equations 3 and 4 can be combined into:

[0046]

[0047] Where X(jω) is a vector composed of the input signals, that is, X(jω) = [X L (jω), X R S(jω)]T, S(jω) is a vector composed of loudspeaker signals, that is, S(jω)=[S L (jω), S R [(jω)]T, C(jω) is the transfer function of the four filters C LL (jω), C RL (jω), C LR (jω) and C RR The matrix H(jω) represents the four room impulse responses H in the frequency domain. LL(jω), H RL (jω), H LR (jω) and H RR The matrix of (jω). Combining equations 5 and 6, we get:

[0048]

[0049] From equation 6 above, we can see that:

[0050]

[0051] In other words, when the filter matrix C(jω) is equal to the inverse H−1(jω) of the room impulse response matrix H(jω) in the frequency domain plus an additional delay τ (at least representing the acoustic delay), then the signal Z reaching the user's left ear... L (jω) is equal to the left input signal X L (jω), and the signal Z reaching the user's right ear. R (jω) is equal to the right input signal X. R (jω), where the signal Z L (jω) and Z R (jω) are respectively related to the input signal X L (jω) and X R (jω) is delayed by the aforementioned delay τ. That is to say:

[0052]

[0053] As can be seen from Equations 7 and 8, from a mathematical perspective, the problem of designing a transear stereo reproduction system is essentially the problem of inverting the transfer function matrix H(jω), which represents the room impulse response (RIR matrix) in the frequency domain. Several methods for matrix inversion are known. For example, the inverse can be determined as follows:

[0054]

[0055] This is the result of applying Cramer's rule to Equation 8 (delay is ignored in Equation 10). The expression adj(H(jω)) represents the adjoint matrix of the RIR matrix H(jω). It can be seen that pre-filtering can be performed in two stages, where the filter transfer function adj(H(jω)) ensures the attenuation of crosstalk, and the filter transfer function det(H)−1 compensates for the linear distortion caused by the transfer function adj(H(jω)). The adjoint matrix adj(H(jω)) always produces a causal filter transfer function, while the compensated filter G(jω)) = det(H)−1 may be more difficult to design. Nevertheless, several known inverse filter design methods may be suitable.

[0056] exist Figure 4 In the example signal flow structure shown, the left ear (signal Z) L The right ear (signal Z) can be considered to be located in the first sound region, and the right ear (signal Z) R () can be considered to be located in the second sound region. Figure 4 The depicted arrangement provides sufficient crosstalk attenuation so that, essentially, the input signal X... L It is reproduced only in the first sound region (left ear), while the input signal X R Reproduced only in the second sound zone (right ear). This concept can be generalized (sound zones are not necessarily associated with the user's ear) and extended to multidimensional cases (more than two sound zones), provided that the system includes at least as many speakers as there are individual sound zones.

[0057] Figure 5 An example audio system is shown in a listening environment such as inside a vehicle having four sound zones 501-504, which can be subjected to signal processing arrangements of one or more embodiments of the present disclosure. Figure 5 The example system shown relies on multiple speakers distributed throughout the vehicle. For example, speakers can be positioned in or around one or more listening locations, such as headrests, dashboard, doors, behind the front seats, and the vehicle headliner. Incoming audio signals are processed, and speaker outputs are controlled to personalize the sound for each area. Figure 5 The diagram illustrates a top view of the vehicle interior 505 from four exemplary listening positions: left front listening position FLP, right front listening position FRP, left rear listening position RLP, and right rear listening position RRP. A stereo signal with left and right channels is reproduced, such that a stereo audio signal is received at each listening position: left front position left channel FLP-LC and right channel FLP-RC, right front position left channel FRP-LC and right channel FRP-RC, left rear position left channel RLP-LC and right channel RLP-RC, and right rear position left channel RRP-LC and right channel RRP-RC. Each channel may include one speaker or a group of speakers of the same or different types, such as a woofer, midrange speaker, and tweeter. The speakers are integrated into the headliner, located on the left and right sides above the listening positions FLP, FRP, RLP, and RRP. Advantageously, the distance between the user's ear and the corresponding speaker should be as short as possible to increase natural speaker isolation between areas. Additionally, the vehicle may be equipped with mid-range and high-frequency speakers (neither shown), typically located at the front, rear, and sides of the vehicle, such as in the dashboard, vehicle floor, door panels, and / or trunk space. The vehicle may also be equipped with multiple microphones (MICn), typically located in the top, front, rear, and sides of the vehicle interior, such as in the dashboard, vehicle ceiling, vehicle door panels, and rear rack.

[0058] Individual sound zone algorithms have been used as a solution for mono audio. One or more embodiments of this disclosure utilize such an algorithm to process the low frequencies of audio by applying a low-pass filter, while preserving the remaining frequency bands (mid to high frequencies) in stereo and / or surround sound, including volume control. A high-pass filter is used to separate the individual components of the audio signal to preserve the stereo / surround sound information to be distributed to the speakers. According to one or more examples, the speakers in the headrest also maintain their stereo configuration, but the low-frequency components are delivered as mono. In this regard, audio content can be played to each zone at different volumes based on the volume control set by the user in each individual sound zone. Ideally, when a user in one zone adjusts their volume, users in other zones will not detect a volume difference in their own individual sound zones.

[0059] 3D audio effects are a set of sound effects that manipulate sound produced by stereo speakers, surround sound speakers, speaker arrays, or headphones. This typically involves virtually placing sound sources at any location in three-dimensional space, not only in front of the user, but also behind, above, or below them. 3D audio (processing) is the spatial domain convolution of sound waves. It is the phenomenon of transforming sound waves (e.g., using head-related transfer function filtering and crosstalk cancellation techniques) to simulate natural sound waves emanating from a point in 3D space. It allows the brain to use the ears and auditory nerves to trick the system into pretending that different sounds are placed in different 3D locations when hearing them, even if the sound is produced by two or more speakers at a fixed location.

[0060] like Figure 6 As shown, upmixing techniques, including source extraction, ambient sound generation, and source allocation, can be used to generate 3D audio. Upmixing is the process of generating additional speaker signals from source material with fewer channels than available speakers, and in many cases, it converts two-channel recordings into multi-channel formats. For example, an upmixing algorithm can be based on describing a stereo recording with two channels L and R as a weighted sum of direct signal sources superimposed with unrelated ambient signals, such as... Figure 6As shown, and which may include the following processing blocks operating in the time or frequency domain: In a translation estimation block 601, which can be used as a source extraction block, the azimuth position or translation coefficient of the original source (i.e., the source in the original (stereo) recording) is estimated, assuming that only one primary source is active at a single time-frequency moment. Translation estimation block 601 outputs, for example, the dry signal DS and the origin direction of the dry signal DS represented by the azimuth signal Ψ. In a direct / ambient sound decomposition block 602, which can be used as an ambient sound generation block, the direct component and ambient sound component are separated using the knowledge of the translation coefficients to provide the ambient sound signal AL for the left channel L and the ambient sound signal AR for the right channel. Based on the estimated signals DS, Ψ, AL, and AR, the original content is remixed in a retranslation block 603, targeting any desired virtual speaker configuration, according to user preferences input to and processed by the retranslation control block 604, which can be used as a source distribution block. The difference between dry and wet signals is that a wet signal is the processed or affected part of the sound, while a dry signal is the original or unaffected part. For example, if a reverb response is used for a vocal track, a wet signal represents the reverberated sound, while a dry signal represents the vocals without reverberation.

[0061] For example, two stereo input channels L and R can be upmixed into a nine-channel signal, with each channel driving one speaker or a group of speakers such as LFCS, LFLS, LFRS, LRLS, LRRS, UFLS, UFRS, URLS, and URRS. Figure 7 As shown. Five speakers or a group of speakers LFCS, LFLS, LFRS, LRLS, LRRS are arranged along a first virtual circle 701 in the lower plane, and the remaining four speakers or a group of speakers UFLS, UFRS, URLS, URRS are arranged along a second virtual circle 702 in the upper plane, which is located above the circle 701 in the lower plane. The user 703 can be positioned at the center of the lower circle 701, with their gaze direction 704 towards the center speaker LFCS.

[0062] Now, only examining the speakers or a group of speakers in the lower plane (LFCS, LFLS, LFRS, LRLS, LRRS), the estimated signals DS, Ψ, AL, and AR can be used according to... Figure 8The signal stream shown creates a stereo-to-five-channel surround sound upmix. The dry signal DS is re-translated across the front speakers LFCS, LFLS, and LFRS using, for example, a vector basis amplitude shift (VBAP) 803 based on the azimuth signal Ψ, while the left and right ambient signals AL and AR are added (using adders 804 and 805) to the signals of the front corner speakers LFLS and LFRS and supplied to the rear corner speakers LRLS and LRRS. To decorrelate the front and rear, short delays 801 and 802 for each ambient signal AL and AR can be included between the front and rear. Alternatively, a more advanced (e.g., time-domain) decorrelector can be used.

[0063] An example of the source extraction process is combined below. Figure 9 The description is provided and detailed in U.S. Patent No. 9,372,251B2, which has been assigned to the assignee of this disclosure and is incorporated herein by reference. Figure 9 An example functional processing block of an audio processing method operating in the frequency domain is shown. This audio processing method includes an audio input signal parsing block 901 and a post-processing block 902. The audio input signal parsing block 901 includes an audio input preprocessing block 903, a sound source vector generation block 904, and a parameter input controller block 905. In other examples, additional or fewer blocks may be used to describe the functionality of the audio processing system.

[0064] exist Figure 9 In this configuration, audio input preprocessing block 903 can receive audio input signal 906. Audio input signal 906 can be a stereo input signal pair, a multi-channel audio input signal (such as a 5-channel, 6-channel, or 7-channel input signal), or any other number of audio input signals greater than or equal to two. Audio input preprocessing block 903 can include any form of time-domain to frequency-domain conversion process. Figure 9 In this example, the audio input preprocessing block 903 includes a windowing block 907 and a converter 908 for each of the audio input signals 906. The windowing block 907 and the converter 908 can perform overlapping window analysis on the time sample blocks and transform the samples using a Discrete Fourier Transform (DFT) or other transform procedures. In other examples, the processing of the audio input signal can be performed in the time domain, and the audio input preprocessing block 903 can be omitted from the audio input signal processing block 901 and can be replaced by a time-domain filter bank.

[0065] A pre-processed (or unprocessed) audio input signal can be provided to a source vector generation block 904. The source vector generation block 904 can generate a source generation vector (Ss). The source vector generation block 904 may include a gain vector generation block 909, a signal classifier block 910, and a vector processing block 911. The gain vector generation block 909 can generate a gain position vector for each of the spatial slices 924. The spatial slice 924 represents the perceived position of a listener in the sound field at a given moment. The listener-perceived sound field includes, for example, a left and right speaker that are typically symmetrical about the center. In other examples, other configurations of the listener-perceived sound field can be implemented, such as a surround sound listener-perceived sound field.

[0066] Generating a gain position vector using gain vector generation block 909 may involve processing using estimated position generation block 912, position filter bank generation block 913, balancing block 914, perceptual model 915, source model 916, and genre detection block 917. Estimated position generation block 912 can calculate estimated perceptual position values ​​using Equation 1 as previously discussed. Position filter bank generation block 913 can calculate position filter banks, and the balancing block can calculate the source generation vector (Ss).

[0067] Perceptual model 915 and source model 916 can be used to improve processing to develop a gain position vector using estimation position generation block 912, position filter bank generation block 913, and balancing block 914. Generally, perceptual model 915 and source model 916 can operate collaboratively to adjust the calculation of the gain position vector on a snapshot-by-snapshot basis to compensate for abrupt changes in the calculated positions of audible sources within the user-perceived sound field. For example, perceptual model 915 and source model 916 can compensate for abrupt changes in the presence and amplitude of a particular sound source in the user-perceived sound field, changes that would otherwise lead to abrupt shifts in perceived position. The perceptual model can perform smoothing on the gain position vector based on at least one of time-based auditory masking estimation and frequency-based auditory masking estimation during the generation of the gain position vector over time (e.g., over multiple snapshots). Source model 916 can monitor the audio input signal and provide smoothing processing to prevent changes in the amplitude and frequency of the audio input signal from exceeding a predetermined rate within a predetermined number of snapshots.

[0068] Considering at least one of the previous snapshots, monitoring can be performed on a bin-by-bin basis for each snapshot or moment of the audio input signal. In one example, two previous snapshots are individually weighted with a predetermined weighting factor, averaged, and used for comparison with the current snapshot. The most recent previous snapshot has a higher predetermined weight than the older snapshot. When the source model 916 identifies a change in amplitude or frequency exceeding a predetermined rate of change, the perception model 915 can automatically and dynamically smooth the gain value in the gain position vector to reduce the rate of change of the perceived position of the sources or audible sounds or audio sources included in the perceived sound field of the audio input signal. For example, when multiple audio sources are sometimes located together in the same perceived position or spatial slice 924 and sometimes occupy different perceived positions at different times, smoothing can be used to avoid the "jumping" phenomenon of audio sources between perceived positions. Such rapid movement between perceived positions could otherwise be perceived by the user as an audio source jumping from a speaker driven by a first output channel to another speaker driven by a second output channel.

[0069] Alternatively or additionally, source model 916 can be used to define the boundaries of a sensing location or spatial slice 924, wherein the sensing location can be automatically adjusted based on the audio sources identified in the audio input signal according to the sources included in source model 916. Therefore, if an audio source is identified as being located at more than one sensing location, the area representing the sensing location can be increased or decreased by adjusting the boundaries of the sensing locations. For example, the area of ​​a sensing location can be widened by adjusting the intersections of filters in a location filter bank (not shown) so that the entire audio source is located at a single sensing location. In another example, if two or more audio sources are determined to be located at the same sensing location, the boundaries of the sensing location or spatial slice 924 can be gradually reduced until the audio source appears in a single spatial slice 924. Multiple audio sources in a single sensing location can be identified, for example, by identifying sources in the source model that correspond to different operating frequency ranges of the identified sources. The boundaries of other spatial slices 924 can also be automatically adjusted. As previously described, the boundaries of sensing locations can overlap, be spaced apart from each other, or be aligned consecutively.

[0070] The perceptual model 915 can also smooth the gain values ​​included in the gain location vector over time to maintain a smooth transition from one moment to the next. The source model 916 can include models of different audio sources included in the audio input signal. During operation, the source model 916 can monitor the audio input signal and adjust the smoothing process together with the perceptual model 915. As an example, the source model 916 can detect the sudden start of a sound source such as a drum and may cause the perceptual model 915 to reduce the amount of smoothing to capture the start of the drum at a unique location in space, rather than blurring it in the spatial slice 924. Using the models included in the source model 916, the perceptual model 915 can take into account the physical characteristics of the sound sources included in the audio input signal when determining how much attenuation should be applied to a given frequency band. Although in Figure 9 In this example, the perceptual model 915 and the source model 916 are shown as separate blocks, but in other examples, the perceptual model 915 and the source model 916 can be combined.

[0071] Genre detection block 917 can detect the genre of the audio input signal, such as classical music, jazz, rock, or talk. Genre detection module 917 can analyze the audio input signal to classify it. Alternatively or additionally, genre detection module 917 can also receive and decode data included in the audio input signal to determine and classify the audio input signal into a specific genre. The genre information determined by genre detection block 917 can also be provided to other blocks in gain vector generation block 909. For example, in surround sound applications, position filter bank generation block 913 can receive an indication that the genre is classical music from genre detection block 917 and automatically adjust the position filter bank by adjusting the crossover points of the filters in the position filter bank to prevent any part of the audio input signal from being output to the right rear and left rear audio output channels.

[0072] Signal classifier block 910 can operate on each of the perceived locations (spatial slices) in the user-perceived sound field to identify one or more audio sources included in a corresponding perceived location. Signal classifier block 910 can identify sound sources based on sound source vectors (Ss). For example, at a first perceived location, signal classifier block 910 can identify the corresponding audio source as a singer's voice; at a second perceived location, the corresponding audio source can be identified as a specific instrument, such as a trumpet; at a third perceived location, multiple corresponding audio sources, such as voice and specific instruments, can be identified; and at a fourth perceived location in the user-perceived sound field, the corresponding audio source can be identified as audience noise, such as applause. The identification of audio sources can be based on signal analysis of audible sounds included in a specific perceived location.

[0073] Signal classifier block 910 can identify sound sources based on input information received from parameter input controller 905, the output signal of vector generation block 909, and / or the output signal of vector processing block 911. For example, given position gain, position vector, and parameters such as RDS data signals provided from parameter input controller 905, identification can be based on the frequency, amplitude, and spectral characteristics of the sound source vector (Ss). Therefore, signal classifier block 910 can perform classification on one or more audio sources included in each of the corresponding perceived locations in the user's perceived sound field. Classification can be based on comparisons, such as comparisons with a predefined library of sound sources, frequencies, or tonal characteristics. Alternatively or additionally, classification can be based on frequency analysis, tonal characteristics, or any other mechanism or technique used to perform source classification. For example, sound source classification can be based on the extraction and / or analysis of reverberation content included in the input signal, estimation using noise included in the input signal, detection of speech included in the input signal, or detection of specific audio sources included in the input signal based on known distinguishing characteristics of audio sources (such as the relatively abrupt start characteristics of a drum).

[0074] Signal classifier block 910 enables vector processing block 911 to assign a given sound source within a given spatial slice 924 to a given output channel. For example, a speech signal can be assigned to a given output channel (e.g., the center output channel) regardless of where the speech signal is located in the user-perceived sound field. In another example, a signal identified as conversational speech (such as talk) can be assigned to more than one output channel to obtain a desired sound field, such as making it more pleasant, improving clarity, or for any other reason.

[0075] exist Figure 9 In this process, the classification of spatial slice 924 can be provided as a feedback audio classification signal to each of the following: position filter bank generation block 913, perceptual model 915, source model 916, and genre detection block 917. The feedback audio source classification signal can include an identifier for each perceptual location in the user's perceived sound field, and an identifier for one or more audio sources included in each perceptual location. Each of the blocks can use the feedback audio source classification signal to perform corresponding processing on subsequent snapshots of the audio input signal.

[0076] For example, the position filter bank generation block 913 can adjust the region of the perceived location by adjusting the position and / or width of the output filters in the position filter bank to capture all or substantially all frequency components of a given sound source within a predetermined number of spatial slices 924 (such as a single spatial slice 924). For example, the position and / or width of the spatial slice 924 can be adjusted by adjusting the crossover points of the filters in the position filter bank to trace and capture identified audio sources (such as audio sources identified as speech signals) in the audio input signal. The perception model 915 can use an audio source classification signal to adjust the masking estimate based on predetermined parameters. Example predetermined parameters include whether the sound source has a strong harmonic structure and / or whether the sound source has a sharp start. The source model 916 can use a feedback audio source classification signal to identify audio sources in the spatial slice 924 of the user's perceived sound field. For example, when the feedback audio source classification signal indicates speech audio sources in some perception locations and music audio sources in other perception locations, the source model 916 can apply speech- and music-based models to different perception locations of the audio input signal.

[0077] The signal classifier block 910 can also provide classification indications for spatial slices 924 on the classifier output line 918. The classification data output on the classifier output line 918 can be in any format compatible with the receiver of the classification data. The classification data may include identifiers of spatial segments 924 and indications of sound sources contained within the corresponding spatial segment 924. The receiver of the classification data can be a storage device, computing device, or any other internal or external device or block with a database or other data retention and organization mechanism. The classification data can be stored in association with other data, such as the audio data for which the classification data was generated. For example, the classification data can be stored in the header or sidechain of the audio data. The classification data can also be used to perform offline or real-time processing on individual spatial slices 924 or all spatial slices 924 in one or more snapshots. Offline processing can be performed by devices and systems with computing capabilities. Once stored in association with audio data, such as in a header or sidechain, the classification data can be used by other devices and systems as part of audio data processing. Real-time processing performed by other computing devices, audio-related devices, or audio-related systems can also use the classification data provided on output line 918 to process the corresponding audio data.

[0078] Genre detection module 917 can use an audio source classification signal to identify the genre of an audio input signal. For example, if the audio source classification signal only indicates speech at different perceptual locations, genre detection module 917 can identify the genre as conversation.

[0079] Gain vector generation block 909 can generate a gain position vector on gain vector output line 919 for receiving by vector processing block 911. Vector processing block 911 can also receive audio input signal 906 as a feedforward audio signal on audio input signal feedforward line 920. Figure 9 In this example, the feedforward audio signal is in the frequency domain. In other examples, the vector processing block 911 can operate in the time domain or a combination of the frequency and time domains, and the audio input signal can be provided to the vector processing block 911 in the time domain.

[0080] Vector processing block 911 can apply a gain position vector to the audio input signal (feedforward signal) in each frequency range to generate a sound source vector (Ss) for each spatial slice 924 in the user-perceived sound field. Individual and independent processing can also be performed on the sound source vectors (Ss) within vector processing block 911. For example, the individual sound source vectors (Ss) can be filtered or their amplitude adjusted before being output by vector processing block 911. Additionally, effects can be added to certain sound source vectors (Ss), such as adding additional reverb to a singer's voice. As part of the processing by vector processing block 911, the individual sound source vectors (Ss) can also be independently delayed, altered, reconstructed, enhanced, or repaired. The sound source vectors (Ss) can also be smoothed or otherwise processed individually before being output by vector processing block 911. Furthermore, the sound source vectors (Ss) can be assembled by vector processing block 911 before output, such as combined or divided. Therefore, the original recording can be "adjusted" based on the level of adjustment of each spatial slice to improve playback quality.

[0081] After processing by vector processing block 911, the processed sound source vector (Ss) can be output as a sound source vector signal on vector output line 921. Each of the sound source vector signals can represent one or more individual audio sources from the audio input signal. The sound source vector signals can be provided as input signals to signal classifier block 910 and post-processing block 902.

[0082] The parameter input controller 905 can selectively provide parameter inputs to the gain vector generation block 909, the signal classifier block 910, and the vector processing block 911. Parameter inputs can be any signal or indication that the blocks can use to influence, modify, and / or improve the processing of the generated gain position vector and / or the processed sound source vector (Ss). For example, in the case of a vehicle, parameter inputs may include external signals such as engine noise, road noise, microphones and accelerometers located inside and outside the vehicle, vehicle speed, climate control settings, the raising or lowering of the convertible roof, the volume of the audio system, RDS data, and audio input signal sources such as optical discs (CDs), digital video decoders (DVDs), AM / FM / satellite radio, cellular phones, Bluetooth connections, MP3 players, or any other audio input signal source. Other parameter inputs may include indications that the audio signal has been compressed by a lossy perceptual audio codec, the type of codec used (such as MP3), and / or the bit rate of the input signal encoding. Similarly, in the case of speech signals, parameter inputs may include indications of the type of speech codec used, its encoding bit rate, and / or indications of speech activity within the input signal. In other examples, any other parameters that are useful for audio processing can be provided.

[0083] In the gain vector generation module 909, parameter inputs can provide information to the genre detection module 917 to detect the genre of the audio input signal. For example, if the parameter input indicates that the audio input signal comes from a mobile phone, the genre detection module 917 can indicate that the audio input signal is a speech signal. Parameter inputs provided to the signal classifier 910 can be used to classify individual audio sources in the spatial slice 924. For example, when the parameter input indicates that the audio source is a navigation system, the signal classifier 910 can look for spatial slices 924 that include speech as an audio source, while ignoring other spatial slices 924. Additionally, the parameters can allow the signal classifier 910 to identify noise or other audio content included in a specific spatial slice 924 with an audio source. The vector processing block 911 can adjust the processing of the spatial slice 924 based on the parameters. For example, in the case of a vehicle, the speed parameter can be used to increase the amplitude of low-frequency audio sources, or certain spatial slices 924, or certain sound source vectors at higher speeds.

[0084] exist Figure 9In this process, the sound source vector signal can be processed by post-processing module 902 to convert the sound source vector signal from the frequency domain to the time domain using a process similar to that of pre-processing module 9. Therefore, post-processing module 902 may include a converter 922 and a windowing module 923 for the sound source vector signal. Converter 922 and windowing block 923 may use Discrete Fourier Transform (DFT) or other transformation processes to convert the time sample blocks. In other examples, different frequency-to-time domain conversion processes may be used. In yet another example, since the processing of sound source vector processing block 904 is performed at least partially in the time domain, the sound source vector signal provided on vector output line 921 may be in the time domain, and post-processing block 902 may be omitted. The sound source vector signal or post-processed sound source vector signal represents an audio source divided into spatial slices 924 and may undergo further processing to drive loudspeakers in a listening space or for any other audio processing-related activities.

[0085] An example of the ambient sound generation process is combined below. Figure 10 and Figure 11 The patent is described in detail in U.S. Patent No. US10728691B2, which has been assigned to the assignee of this disclosure and is incorporated herein by reference. Figure 10 An example process for generating ambient sound 209 by creating a virtual space within a listening environment (such as the interior of a vehicle) is described. Audio source 1001 is configured to provide an incoming audio signal to a vehicle audio controller (not shown). Audio source 1001 can be any of an FM radio station, AM radio station, high-definition (HD) audio radio station, satellite broadcast provider, input from a mobile phone, input from a tablet computer, etc. A user can select the appropriate audio source 1001 via a source selector 1002 located on the vehicle audio controller or via a user interface (not shown) that can be implemented anywhere on the vehicle or mobile device (not shown).

[0086] Reverb extraction block 1003 (or extraction block 1003) removes reverb from the incoming audio signal to provide a dry audio signal. This operation is performed to prepare the incoming audio signal to receive the corresponding reverb effect for the selected location. It should be understood that the reverb extraction block 1003 may not be able to completely remove reverb from the incoming audio signal, and some reverb residue may still remain on the dry audio signal. Stereo equalizer block 1004 receives the dry audio signal from the reverb extraction block 1003. Stereo equalizer block 1004 can be used as a conventional stereo equalizer in a vehicle and is configured to equalize the incoming audio signal for user playback.

[0087] The virtual site process receives signals from each corresponding microphone in the vehicle (such as...). Figure 3The microphone MICn is shown as the input. The audio captured by the microphone M can correspond to music, speech, and ambient noise inside the vehicle. Microphone equalization block 1005 receives the captured audio from the microphone MICn and equalizes the captured audio (i.e., enhances or attenuates the energy of individual frequency bands). Feedback equalization block 1006 receives the output from microphone equalization block 1005. Process 209 also includes delay block 1007, audio mixer 1008, and spider reverb block 1009. Delay block 1007 receives dry audio from extraction block 1003 to time-align the dry audio with the captured audio from the microphone MICn. This situation takes into account the ambient sound generation process 209 (see Ambient Sound Generation Process 209). Figure 2 This involves handling the delay of the incoming audio signal. It is desirable to ensure that the playback of entertainment data regarding the incoming audio signal is time-aligned with the audio signal captured by the microphone MICn. Consider an example where a vehicle occupant claps or sings along with the entertainment data of the incoming audio signal. In this case, it is desirable to time-align the playback of the entertainment data regarding the incoming audio signal with the clapping or vocal input from the vehicle occupant (captured by the microphone MICn) for playback. By capturing the playback of the entertainment data of the incoming audio signal and the clapping or vocal input (or other actions performed by the vehicle occupant consistent with the entertainment data) by the microphone MICn, this aspect further provides the vehicle occupant with the experience of being in the desired location, just as one expects to hear noise to some extent consistent with the audio playback of the location, including the audience. Therefore, by capturing the ambient noise inside the vehicle with the microphone MICn and combining this data with the entertainment data of the incoming audio signal and subsequently adjusting the reverberation of the mix, this aspect enhances the vehicle occupant's experience and provides the vehicle occupant with the feeling of being in the desired location.

[0088] Delay block 1007 may or may not apply a delay, depending on the processing speed of the process. Mixer 1008 is configured to mix the reverberation in the audio captured by microphone MICn with any remaining reverberation in the incoming audio signal. Mixer 1008 receives the signal WINDOW / CONVERTIBLE STATUS, which indicates whether the windows, convertible roof, or sunroof is open or closed. Similarly, if the windows, convertible roof, or sunroof is open and there is too much noise in the signal, mixer 1008 can mute the signal captured from microphone MICn. Likewise, mixer 1008 controls how much noise or voice data (i.e., audio data captured from multiple microphone MICn) inside the vehicle is fed back to spider reverberation block 1009, and how much audio is fed to spider reverberation block 1009. Generally, mixer 1008 determines the fusion of the audio captured at microphone MICn with direct audio (or dry audio) to achieve the desired fusion.

[0089] User interface 1010 provides control signals to process 209, instructing process 209 to play audio in a selected location (or virtual location). As described above, the selected location can correspond to any of the listening environments of a stadium, concert hall (e.g., large, small, or medium-sized), recording studio, and vehicle interior, which differ from the listening environment of the user's vehicle. Spider reverb block 1009 receives output from mixer 1008, which corresponds to the mixed dry audio and captured audio. Spider reverb block 1009 typically includes multiple spider reverb blocks 1011a-1011n (or "1011") and multiple location equalizer blocks 1012a-1012n (or "1012"). Generally, each spider reverb block 1011 and its corresponding location equalizer block 1012 adds or adjusts the amount of reverberation on the output of mixer 1008 to provide the user with a selected or desired location. Specifically, spider reverb block 1011 replicates different reverberation characteristics of different walls for the selected location. Spider reverb block 1009 adjusts the reverberation to correspond to a specified or selected location, and location equalization block 1012 controls the brightness characteristics of the walls inside the vehicle to provide the desired brightness characteristics for the selected location. The selected location could correspond to a stadium, a large concert hall, a medium-sized concert hall, etc. For example, if a user selects process 209 to play audio as if the user were in Carnegie Hall, then spider reverb block 1011 is configured to provide a reverberation effect on the walls inside the vehicle, making it sound like the walls of Carnegie Hall. Therefore, this makes the user feel as if he / she is actually sitting in a vehicle listening to audio from Carnegie Hall. Process 209 includes storing any number of desired locations in memory (not shown) and also taking into account various front, side, rear, and top walls of the selected location, as well as how audio reflects or reverberates from such surfaces of the walls. For example, the memory may include storing various preset frequency values ​​corresponding to the characteristics of the walls in a particular location, and the location equalizer 1012 may enhance or reduce the frequency levels of the audio output by the mixer 1008 and the spider reverb block 1011 to further enhance the user’s feeling that they are actually in the corresponding or selected location.

[0090] For example, consider a scenario where a selected location typically features a low ceiling made of metal and a carpeted outer wall. The ceiling can have very loud and fast reflection characteristics compared to other walls where sound would be very muffled and reflection times are slow. A spider-style reverb block 1009 adjusts the reverberation of the incoming and captured audio signals to provide the desired location, and a corresponding location equalization block 1012 controls the equalization of the incoming and captured audio signals to simulate playback in the desired location and the brightness characteristics of the walls in the desired location. Generally, the speakers in a vehicle collectively provide outputs corresponding to the desired location, and corresponding speakers in a given wall can each receive discrete inputs to simulate the desired brightness characteristics of that given wall in the desired location. For example, speakers in the vehicle's ceiling can receive an equalized output to provide the appearance of sound bouncing from the ceiling with a fast reflection time, consistent with the low ceiling of the selected location as described above. Similarly, the equalization can be adjusted differently for each audio output provided to a corresponding speaker in a particular wall to match the individual walls in the selected location.

[0091] Speaker equalizer 1013 receives output from spider reverb block 1009 to provide a more uniform audio response within the vehicle. Speaker equalizer 1013 compensates for issues with the speakers in the vehicle. If the user selects to listen to the incoming audio in normal mode, mute block 1014 is provided to simply remove the amount of reverberation added by spider reverb block 1009. User interface 1010 can transmit a signal indicating a request to vehicle audio controller 26 to disable the reverberation effect added to achieve the selected location. In response to this request, process 209 can activate mute block 1014 to simply disable the playback of audio in the selected location. Adder 1015 receives output from spider reverb block 1009 (or from mute block 1014) and also from stereo equalizer 1004, and adds these two audio inputs together to provide a virtual location output signal VVS to, for example, source distribution block 210.

[0092] Figure 11 Generally shown for use Figure 10 The illustrated process structure describes a method for generating a virtual space within a listening room. Combined with... Figure 11 The operations described can be performed in any order, and it should be noted that various operations can be performed simultaneously. The order of operations performed may vary depending on the specific implementation.

[0093] In operation 1101, process 209 receives an input audio signal from audio source 1001. As described above, audio source 1001 may correspond to any one of an FM radio station, a high-definition (HD) audio station, a satellite broadcasting provider, a mobile phone input, a tablet input, an MP3 player, or any other source that provides entertainment data in conjunction with the provided audio signal. Generally, the incoming audio signal may correspond to audio data to be played to entertain the occupants of the vehicle.

[0094] In operation 1102, process 209 removes reverberation from the incoming audio signal to provide a dry audio signal.

[0095] In operation 1103, process 209 receives captured audio signals from each microphone (MICn) inside the vehicle. For example, a vehicle audio controller, such as... Figure 1 The audio processing system shown enhances or reduces the energy of each frequency band of the captured audio signal. As mentioned above, the captured audio signals typically correspond to music, noise captured from vehicle occupants (which corresponds to entertainment data in the incoming audio signal, including entertainment data from electronic audio sources), speech (or conversations of vehicle occupants), and / or ambient noise entering the vehicle from outside, ambient noise from inside the passenger compartment, etc.

[0096] In operation 1104, processing unit 209 equalizes each captured audio signal. For example, process 209 enhances or weakens the energy of individual frequency bands in the captured audio signal.

[0097] In operation 1105, process 209 may optionally employ a time delay or delay in the transmission of the dry audio signal and the captured audio signal to ensure that the playback of entertainment data on the dry audio signal is consistent with the captured audio signal.

[0098] In operation 1106, process 209 determines whether any one of the windows, convertible roof, and sunroof is open. If process 209 determines that any one of the windows, convertible roof, and sunroof is closed, the method moves to operation 1107. If process 209 determines that any one of the windows, convertible roof, and sunroof is open, the method moves to operation 1108.

[0099] In operation 1107, process 209 mixes the reverberation on the captured audio signal with the dry audio signal to achieve the desired fusion of noise, music and / or speech information on the captured audio signal with entertainment data on the incoming audio signal received from audio source 1001.

[0100] In operation 1108, process 209 mutes the captured audio signal because such a signal would carry too much noise (e.g., ambient noise, such as wind, road noise, etc.) if one of the windows, convertible roof, or sunroof is open. In this case, the process in operation 1109 can simply adjust the reverberation of the incoming audio signal from audio source 1001 to play back the incoming audio signal in a selected location inside the vehicle.

[0101] In operation 1109, process 209 receives a control signal indicating a desired location to be simulated inside the vehicle during audio playback.

[0102] In operation 1110, process 209 adjusts the reverberation of the mixed captured audio signal and the dry audio signal to play entertainment data from the incoming audio signal from audio source 1001 in selected locations within the vehicle. Additionally, process 209 equalizes the frequencies of the mixed captured audio signal and the dry audio signal to provide the desired brightness characteristics to the individual walls of the selected locations.

[0103] The following is combined with Figure 12 An example describing the user preference handling process is provided. Figure 12 The illustrated user preference handling process employs a graphical user interface (GUI) 1201, which includes, for example, a touchscreen 1202 for visual / tactile interaction with the user, at least one graphics processing unit (GPU) 1203 for driving and controlling the touchscreen 1202, and an image generator 1204 that provides not only visual user guidance but also a graphical representation of the listening room or sound area selected by the user to be displayed on the touchscreen 1202. For example, the user can use the touch function of the screen 1202 to move the graphical representation of the extracted source to the desired position in the graphical representation of the listening room displayed using the visual function of the screen 1202. This representation is generated by the image generator 1204. The graphics processor 1203 converts the position of the graphical representation of the source extracted on the screen 1202 into position data 1207 in the listening room.

[0104] At least one processor 1205 converts location data 1207 into control signals 1208 for controlling acoustically relevant parameters, such as source distribution control signals and ambient sound control signals, which control parameters such as filter parameters, delay times, and filter interconnection structures. To convert location data 1207 into control signals 1208 for controlling acoustically relevant parameters, artificial intelligence (such as a machine learning (ML) algorithm 1206) can be employed, which assigns certain desired locations of virtual sources to certain auditory impressions. The ML algorithm 1206 can be trained on a variety of audio signals and with a large number of users. Figure 12The illustrated user preference handling process maps user preferences input into the touchscreen 1202, related to the output virtual source distribution, to parameters suitable for 3D rendering. Although in Figure 12 The functions shown are presented as separate blocks, but some of the functions shown can be combined into a block, and all functions other than those of the touch screen 1202 can also be combined into a block, for example, as in the graphics processor unit 1203.

[0105] Figure 13 A possible workflow for implementing the above-described sound reproduction method is shown, wherein the method includes at least the following operations: (a) receiving one or more input audio signals representing audio content to be reproduced; (b) generating at least two sound regions using a plurality of speakers configured, positioned, and operated to spatially limit the audibility of the audio content to be reproduced to one of the at least two sound regions; and (c) reproducing the audio content to be reproduced in the one of the at least two sound regions using at least some of the plurality of speakers, wherein the at least some of the plurality of speakers are configured, positioned, and operated to generate sound from the input audio signals capable of producing a three-dimensional audio effect, the three-dimensional audio effect comprising placing a virtual sound source at any location in a three-dimensional space, and the three-dimensional space being one of the at least two sound regions. These operations (a)–(c) can be performed in any order, and it is intended that the various operations can be performed simultaneously with each other. The order in which the operations are performed may vary depending on the requirements of a particular implementation. Figure 13 Here is an example implementation of this approach, including the following program:

[0106] Program 1301: The user preference processing block inputs user preferences regarding the sound region to be selected and the desired source distribution, and maps them to applicable control information. The user can use an interface, such as a graphical user interface (GUI), through which he / she can select, for example, sounds and instruments.

[0107] Program 1302: Receives and processes audio signals from the mixer block of the selected sound region. Program 1302 includes subroutines 1303-1305:

[0108] Subroutine 1303: The source extraction block extracts the source from the input audio signal based on information from the user preference processing block.

[0109] Subroutine 1304: The ambient sound generation module generates ambient sound based on the input signal.

[0110] Subroutine 1305: The source distribution block derives the translation coefficients of the source location based on the user input via the control block and provides the final signal distribution.

[0111] Program 1306: The sound field processing block establishes and controls the sound region, including the selected sound region.

[0112] Program 1307: Reproduce the altered audio signal in the selected sound region.

[0113] Under typical conditions inside a vehicle, the upmixing rendering process can be merged with the sound zone process, where each sound zone is focused on a specific location, such as a seat, thereby enriching the personalized immersive audio experience for all passengers. The input mono or multi-channel audio signal is processed by an upmixer block within a sound zone. A source extraction block extracts the source from the input audio signal based on information from a user preference evaluation block. An ambient sound generation block generates ambient sound based on the data signal. A source distribution block derives translation coefficients for source localization based on user input via a control block and is responsible for the final signal distribution. The user can input his / her preferences via an interface (e.g., a visual interface), where he / she can adjust the position of vocals and instruments, for example, on a (touch) screen. The interface generates corresponding virtual source distribution parameters based on the desired source distribution depicted on the screen, and the source distribution block generates a 3D virtual source distribution based on these parameters. Furthermore, this invention proposes enhancing the user experience by using an intelligent source distribution algorithm to further enhance "personalization."

[0114] This allows every user to experience 3D audio rendering in the most likely way possible, whether in a vehicle or any other listening room, which is currently not the case. Existing technologies focus on dedicated listening positions or optimal listening positions, while the proposed idea overcomes this limitation by combining 3D audio technology with sound regions. All users can enjoy a personalized 3D audio experience, not limited to an "optimal listening position," and the rendering is tailored to dedicated sound regions. Audio rendering is further enhanced by involving end-users in source distribution using intelligent source panning. Each user can listen to immersive audio rendering in their own unique way.

[0115] The embodiments have been described for purposes of illustration and description. Suitable modifications and changes to the embodiments can be made in light of the foregoing description or by means of practice. For example, unless otherwise stated, one or more of the described methods can be performed by suitable means and / or combinations of means. The described methods and associated actions can be performed in various orders other than in the order, in parallel, and / or simultaneously as described in this application. The described system is exemplary in nature and may include additional and / or omitted elements.

[0116] As used in this application, elements or steps described in the singular and preceded by the words "an" or "a" should be understood to not exclude multiple said elements or steps, unless such exclusion is specified. Furthermore, references to "an embodiment" or "an example" in this disclosure are not intended to be construed as excluding the existence of additional embodiments that also incorporate the described features. The terms "first," "second," and "third," etc., are used merely as illustrative marks and are not intended to impose numerical requirements or a particular order on their objects.

[0117] While various embodiments of the invention have been described, it will be apparent to those skilled in the art that many embodiments and implementations can be made within the scope of the invention. Specifically, those skilled will recognize the interchangeability of various features from different embodiments. Although these techniques and systems have been disclosed in the context of certain embodiments and examples, it should be understood that these techniques and systems can be extended from the specifically disclosed embodiments to other embodiments and / or uses and their apparent modifications.

Claims

1. A method for sound reproduction, comprising: Receives an input audio signal representing the audio content to be reproduced; At least two sound zones are generated using multiple loudspeakers, which are configured, positioned, and operated to spatially limit the audibility of the audio content to be reproduced to one of the at least two sound zones; as well as The audio content to be reproduced is reproduced in one of the at least two sound regions using at least some of the plurality of speakers, wherein at least some of the plurality of speakers are configured, positioned and operated to generate sound from an input audio signal that produces a three-dimensional audio effect, the three-dimensional audio effect including placing a virtual sound source at any location in a three-dimensional space, and the three-dimensional space is one of the at least two sound regions.

2. The method of claim 1, wherein reproducing the audio content includes analyzing the received input audio signal to identify the original sound source therein and its corresponding original position, and extracting the dry audio signal of the original sound source and the corresponding position signal of the original sound source from the input audio signal.

3. The method of claim 2, wherein extracting the dry audio signal of the original sound source and the corresponding position signal of the original sound source includes extracting at least one of the center dry signal, human voice dry signal, instrument dry signal, dry residual signal and the corresponding position signal of the original sound source.

4. The method of claim 2 or 3, wherein reproducing the audio content includes receiving an ambient sound control signal representing a desired ambient sound to be reproduced, generating an ambient sound signal based on the ambient sound control signal, and adding the ambient sound signal to the sound to be reproduced.

5. The method of claim 4, wherein generating the ambient sound signal includes at least one of generating early reflections and generating reverberation of the virtual sound source.

6. The method of any one of claims 2 to 5, wherein reproducing the audio content comprises: Receive the source distribution control signal representing the desired virtual source distribution; as well as Based on the source distribution control signal and the dry audio signal of the original sound source, a virtual sound source at the virtual sound source position in the sound to be reproduced is generated from the dry audio signal of the original sound source and the corresponding position signal of the original sound source.

7. The method of claim 1, wherein the input audio signal has multiple channels, and reproducing the audio content includes upmixing the input audio signal to a number of channels greater than the number of channels of the audio input signal.

8. The method of claim 7, wherein the supermixing comprises at least one of the following: The received input audio signal is analyzed to identify the original sound source and its corresponding original position, and the dry audio signal of the original sound source and the corresponding position signal of the original sound source are extracted from the input audio signal. Generate an ambient sound control signal representing the desired ambient sound to be reproduced, and generate an ambient sound signal based on the ambient sound control signal; and Based on the source distribution control signal, a virtual sound source at the virtual sound source location is generated from the dry audio signal of the original sound source and the corresponding position signal of the original sound source.

9. The method of any one of claims 2 to 8, further comprising receiving user preferences via a user interface and generating at least one of a source distribution control signal and an ambient sound control signal based on the received user preferences.

10. The method of claim 9, wherein generating at least one of a source distribution control signal and an ambient sound control signal based on received user preferences comprises performing a machine learning-based algorithm that maps user preferences associated with the virtual source location to the source distribution control signal.

11. The method of claim 9 or 10, wherein the user preference is received from a graphical user interface, the graphical user interface generating location data corresponding to the desired location of the virtual source.

12. The method of any one of claims 1 to 11, wherein generating the at least two sound regions comprises performing matrix inverse filtering on the signal to be reproduced by the plurality of loudspeakers.

13. A sound reproduction system comprising a plurality of loudspeakers, a multi-channel amplifier configured to drive the plurality of loudspeakers, and at least one processor configured to drive the amplifier, the at least one audio signal processor being configured to execute instructions of a computer program that, when executed by the at least one processor, causes the at least one processor to perform the method as claimed in any one of claims 1 to 11.

14. The system of claim 13, wherein at least some of the plurality of speakers are disposed around one of at least two sound zones.

15. The system of claim 13 or 14, further comprising a touchscreen.

Images