Audio processing method, apparatus and system, vehicle, storage medium and program product

Abstract

The application provides an audio processing method, device and system, vehicle, storage medium and program product; the audio processing method comprises: determining key information of original audio data, the key information comprising a plurality of sound source objects, category information and reverberation information, the category information representing a genre to which the original audio data belongs and / or an emotion of the original audio data; generating a multi-channel pseudo-ambisonics code stream based on the plurality of sound source objects, the category information and the reverberation information; determining driving signals of a plurality of target loudspeakers in a current use scenario based on acoustic parameters corresponding to the current use scenario and the multi-channel pseudo-ambisonics code stream; and distributing the driving signals of each target loudspeaker to a corresponding loudspeaker channel for playing. The scheme separates sound source objects, genres, emotions, reverberation and the like, re-upsamples in combination with an actual use scenario, and reconstructs a sound field of the actual use scenario, thereby providing higher-quality, personalized and immersive sound effects.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of audio technology, and in particular to an audio processing method, apparatus and system, vehicle, storage medium and program product. Background Technology

[0002] For audio applications (such as car audio systems, home theaters, cinemas, etc.), fixed acoustic models or simple sound image shifting rules are mainly used for upmixing, and fixed equalizer settings are used for very simple surround sound processing. This results in problems such as unnatural sound field, blurred sound field, false sense of space, chaotic instrument positioning, and poor listening experience. Summary of the Invention

[0003] One objective of this application is to provide an audio processing method to solve problems such as unnatural sound field, blurred sound field, false spatial sense, chaotic instrument positioning, and poor auditory experience in audio usage scenarios in related technologies; another objective is to provide an audio processing system; a third objective is to provide a vehicle; a fourth objective is to provide a computer-readable storage medium; and a fifth objective is to provide a computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an audio processing method, the method comprising: Determine key information of the original audio data, including multiple sound source objects, category information, and reverberation information. The category information represents the genre to which the original audio data belongs and / or the emotion of the original audio data. Based on the multiple sound source objects, the category information, and the reverberation information, a multi-channel pseudo-immersive sound bitstream is generated; Based on the acoustic parameters corresponding to the current usage scenario and the multi-channel pseudo-immersive sound bitstream, determine the driving signals of multiple target loudspeakers in the current usage scenario; The drive signal for each target loudspeaker is assigned to the corresponding loudspeaker channel for playback.

[0005] In this embodiment, firstly, different sound source objects, genres, emotions, reverb, etc., are separated from the original audio data, achieving a deep understanding of the audio content and greatly improving the separation quality. This lays the foundation for developing differentiated rendering strategies for audio of different styles, enriching the processing effects and reducing the possibility of unnatural sound fields due to low separation quality. Secondly, a panoramic sound bitstream is generated based on the information of each sound source object, genre, emotion, and reverb. In other words, it can upmix two-channel stereo audio to output high-quality multi-dimensional panoramic sound audio with immersion, clear positioning, and content adaptability. This enriches the mixing processing methods, allows for customized adjustments based on the specific characteristics of the audio content, improves the mixed effect, and reduces the possibility of blurred sound fields, false spatial sense, and chaotic instrument positioning caused by rigid or monotonous mixing processing methods, achieving a truly immersive high-quality listening experience. Finally, based on the acoustic parameters of the current usage scenario (such as vehicles, home theaters, etc.) and the panoramic sound bitstream, the driving signals of each speaker are determined and output. In other words, it can accurately reconstruct the sound field according to the characteristics of the audio and the usage environment, providing listeners with a higher quality and more personalized sound effect, thereby greatly improving the listening experience.

[0006] In some implementations, determining the key information of the original audio data includes: using a pre-trained separation model to separate the original audio data to obtain the multiple sound source objects; and analyzing the original audio data to obtain the category information and the reverberation information.

[0007] In this application's implementation, on the one hand, a trained separation model is used to separate the various sound source objects in the original audio data, improving separation efficiency and accuracy while significantly reducing processing costs. Furthermore, since the pre-trained separation model possesses stronger generalization ability and multi-scenario adaptability, it enhances the application capability of this method (e.g., simple or complex real-world audio environments). On the other hand, analyzing and processing the original audio data to obtain reverberation, category, and emotion not only facilitates subsequent inference of the acoustic environment and artistic intent based on reverberation, achieving the goal of realistically recreating the listening experience, but also deeply explores the acoustic and semantic value of sound. This enables precise decision-making and experience upgrades in fields such as music production, intelligent interaction, and psychological care, thereby not only improving the automation level of audio processing but also making human-computer communication more humane.

[0008] In some implementations, the separation model includes an encoding module, a separation module, and a decoding module; the step of using a pre-trained separation model to separate the original audio data to obtain the multiple sound source objects includes: preprocessing the original audio data to obtain a complex spectrum corresponding to the original audio data; using the encoding module to encode the complex spectrum to obtain a first feature; using the separation module to perform time-frequency domain modeling on the first feature to obtain a second feature; and using the decoding module to decode the second feature to obtain the multiple sound source objects.

[0009] In this embodiment, firstly, the original time-domain mixed signal is converted into a complex spectrum, which simultaneously preserves amplitude and phase information, achieving a more complete and natural speech reconstruction and significantly improving listening quality and processing accuracy. Compared to traditional methods that only process the amplitude spectrum, complex spectrum modeling allows the method to understand the spatiotemporal structure of sound, not just volume. Secondly, the high-dimensional features of the complex spectrum are extracted through an encoding module. This not only deeply integrates the joint representation of amplitude and phase, fully preserving the complete physical meaning of the waveform and significantly improving the accuracy and naturalness of the task, but also uncovers high-dimensional hidden patterns (such as capturing the coherence relationship between different frequencies and the phase evolution law between time and frequency), enhancing the modeling ability for complex acoustic scenes. Compared to methods that only process the amplitude spectrum or low-dimensional features, this approach fully releases the expressive potential of complex domain data, enabling the model to better distinguish overlapping speech and separate reverberation components. Furthermore, by performing time-frequency domain modeling on high-dimensional features through a separation module, the complex signal is decomposed into two collaborative but independent processing paths. This fully leverages the complementary advantages of the time and frequency domains, achieving joint modeling of "local dynamics" and "global structure," significantly improving accuracy, robustness, and interpretability. Finally, the decoding module reconstructs the modeled features into a time-domain signal, enabling high-fidelity, low-distortion speech reconstruction. This significantly improves the naturalness of the sound and the practicality of the task, ensuring that the reconstructed speech is closer to real human voices in terms of detail, rhythm, and phase.

[0010] In some implementations, the complex spectrum is used to characterize the frequency range of the speech band of the original audio data; the encoding process of the complex spectrum to obtain the first feature includes: dividing the frequency range of the speech band using a preset sparse banding strategy to obtain multiple first sub-bands; and performing downsampling compression processing on each first sub-band at a corresponding multiple to obtain the first feature.

[0011] In this embodiment, a sparse banding strategy is adopted to decompose the audio band into multiple sub-bands, and different sub-bands are downsampled and compressed at different ratios. Banding is performed according to the importance of speech to prioritize the preservation of key frequency bands. At the same time, differentiated downsampling compression is used to optimize the computational load and transmission efficiency. This achieves the goal of allocating most of the computing power to the low and mid frequencies that are sensitive to the human ear while compressing the high-frequency dimension. This not only reduces the frequency dimension and computational complexity of the model with almost no loss of model accuracy, but also extracts high-dimensional features suitable for separation tasks, achieving the optimal balance between preserving key information and compressing redundant data.

[0012] In some embodiments, the separation module includes a frequency domain processing unit, a weighted processing unit, and a time domain processing unit; the step of using the separation module to perform time-frequency domain modeling on the first feature to obtain a second feature includes: using the frequency domain processing unit to perform frequency band modeling on the first feature to obtain a third feature; using the weighted processing unit to perform weighted processing on the first feature and the third feature to obtain a fourth feature; using the time domain processing unit to perform frame number modeling on the fourth feature to obtain a fifth feature; and using the weighted processing unit to perform weighted processing on the fourth feature and the fifth feature to obtain the second feature.

[0013] In this embodiment, firstly, frequency domain processing units are used to perform frequency band modeling on high-dimensional features, which can accurately reveal the global periodic structure and frequency fingerprint (i.e., the harmonic distribution and energy concentration areas of the signal), enhance the ability to distinguish sound sources, enable the model to clearly hear the essential composition of sound, and significantly improve robustness and efficiency. Secondly, frame-by-frame modeling is performed on the weighted features through time domain processing units. Since the weighted high-dimensional features have already incorporated key frequency band information, but the smoothness and dynamic changes of their time axis still need to be finely depicted, frame-by-frame modeling can accurately capture local dynamic changes and temporal continuity, ensuring fluent and natural speech and efficient recognition of instantaneous events. This is a key step in achieving the leap from clear hearing to true hearing. Finally, by using a weighted processing unit to weight the encoded features and the features after frequency band modeling, as well as the features after frequency band modeling and the features after frame number modeling, the goal of dynamically weighting for different times and frequency positions is achieved. This realizes differentiated attention to the importance of music features. That is, for spectrally rich segments (such as the chorus) or key frequency ranges, the model will automatically assign higher weights to enhance the priority of these regions in the separation process, thereby improving the effectiveness of music source separation. At the same time, it can also capture multi-dimensional dependencies and importance in the input sequence, further improving the model's ability to represent and process complex music data.

[0014] In some implementations, the decoding process of the second feature to obtain the plurality of sound source objects includes: dividing the frequency range of the second feature using a preset sparse banding strategy to obtain a plurality of second sub-bands; and performing upsampling expansion processing on each of the second sub-bands at a corresponding multiple to obtain the plurality of sound source objects.

[0015] In the embodiments of this application, a sparse banding strategy is adopted to decompose the features after time-frequency domain modeling into multiple sub-bands, and different ratios of upsampling expansion are applied to different sub-bands. This can achieve priority modeling of key information by dividing the bands according to their importance, while differentiated upsampling expansion optimizes the allocation of computing resources, enabling the model to allocate attention and computing resources more intelligently under limited computing power, and achieving synergistic optimization of key frequency band focusing and on-demand resource allocation.

[0016] In some embodiments, the method further includes: inputting training samples into a separation model to be trained to obtain multiple predicted sound source objects; determining a target loss value based on the predicted multiple sound source objects and the label information corresponding to the training samples; and updating the model parameters of the separation model at least once based on the target loss value to obtain the trained separation model.

[0017] In this embodiment, training the separation model using training samples and their label information significantly improves the model's convergence speed, separation accuracy, and generalization ability, especially in scenarios with small sample sizes, effectively mitigating the risk of overfitting. Furthermore, compared to unsupervised pre-training, supervised training allows the model to learn task-related discriminative features in the initial stage, laying a solid foundation for subsequent fine-tuning.

[0018] In some embodiments, the step of analyzing and processing the original audio data to obtain the category information and the reverberation information includes one of the following: inputting the original audio data into a pre-trained first analysis model to obtain the category information and the reverberation information; or inputting the original audio data and the plurality of sound source objects into a pre-trained second analysis model to obtain the category information and the reverberation information.

[0019] In the embodiments of this application, on the one hand, by using a trained first analysis model to analyze the category, emotion, reverberation information, etc. of the audio, end-to-end joint analysis of multi-dimensional acoustic attributes can be achieved, significantly improving analysis efficiency, consistency, and contextual relevance. Compared with step-by-step independent processing, joint modeling avoids error accumulation, reduces information loss and error propagation, and can capture the implicit relationship between category, emotion, and acoustic environment. On the other hand, by using a trained second analysis model to analyze the category, emotion, reverberation information, etc. of the audio by integrating each object track within the audio, refined semantic analysis in multi-sound source scenarios can be achieved, significantly improving the accuracy of analysis, spatial awareness, and contextual consistency. Compared with analysis based solely on mixed signals, secondary modeling using separated "object tracks" (such as independent speakers and background sound events) can avoid sound source interference and accurately locate the attribute characteristics of each track.

[0020] In some implementations, generating a multi-channel pseudo-immersive audio bitstream based on the plurality of sound source objects, the category information, and the reverberation information includes: determining a sound field effect model based on the reverberation information, the category information, and the plurality of sound source objects; wherein the sound field effect model includes global sound field attributes, spatial attributes of each of the sound source objects, and processing strategies for each of the sound source objects; generating a base sound field bitstream based on the global sound field attributes and the original audio data; generating an object bitstream based on the spatial attributes of each of the sound source objects and the processing strategies for each of the sound source objects; and generating the multi-channel pseudo-immersive audio bitstream based on the base sound field bitstream and the object bitstream.

[0021] In this embodiment, firstly, by making fusion decisions on reverberation, category, and various sound source objects, a unique sound field psychological model is dynamically generated for the audio, achieving multi-dimensional perception fusion, constructing an emotionally resonant sound field, optimizing cognitive comfort, and enabling deep personalization and contextual adaptation of the listening experience, significantly improving the emotional resonance, scene immersion, and cognitive comfort of the audio content. Secondly, a base sound field bitstream is generated using global sound field attributes and raw audio data to simulate the target virtual acoustic environment, providing a sense of spatial enclosure for the entire sound field. Thirdly, object bitstreams are generated using the spatial attributes and processing strategies of each object track, enabling efficient compression and flexible reconstruction of sound field information, significantly improving bandwidth utilization, spatial immersion, and terminal adaptability of audio transmission. Finally, by mixing the base sound field bitstream and object bitstreams, collaborative processing of environmental perception and object rendering is achieved, thereby maintaining clear sound image localization while enhancing spatial enclosure.

[0022] In some implementations, determining the sound field effect model based on the reverberation information, the category information, and the plurality of sound source objects includes: determining the virtual acoustic environment corresponding to the original audio data based on the reverberation information; and determining the sound field effect model from a preset rule base based on the category information, the plurality of sound source objects, and the virtual acoustic environment.

[0023] In the embodiments of this application, on the one hand, by inferring the most likely virtual acoustic environment of the audio based on reverberation information, inverse modeling of sound field perception can be achieved, significantly improving the spatial realism, rendering adaptability, and interactive immersion of audio content. Compared to environmental assumptions that rely solely on the original waveform or metadata, inverse inference based on reverberation features can accurately restore the acoustic context, providing a physical basis for virtual reconstruction and intelligent processing. On the other hand, by dynamically selecting appropriate sound field effect models based on emotion, genre, object tracks, and the virtual acoustic environment, a deep integration of personalized, contextualized, and immersive acoustic experiences can be achieved, significantly improving the emotional expressiveness, style fidelity, and spatial realism of audio content. Compared to static and uniform sound field processing, the dynamic adaptation strategy allows the sound to truly change according to the person, the emotion, and the scene.

[0024] In some implementations, generating a base sound field bitstream based on the global sound field attributes and the original audio data includes: determining a reverberation processing strategy and an equalization processing strategy based on the global sound field attributes; performing reverberation processing on the original audio data according to the reverberation processing strategy to obtain first audio data; and performing equalization processing on the first audio data according to the equalization processing strategy to obtain the base sound field bitstream.

[0025] In the embodiments of this application, global environmental reverberation and equalization processing are applied according to the global sound field properties of the sound field effect model, which can better simulate the sound propagation characteristics under different environments, form a realistic base sound field bitstream, and provide a strong and realistic sense of spatial immersion for the entire sound field, thereby improving the realism and immersion of the hearing.

[0026] In some implementations, the object bitstream includes a rendered bitstream of the speaker channel corresponding to each of the sound source objects; generating the object bitstream based on the spatial attributes and processing strategy of each of the sound source objects includes: for each sound source object, determining the co-processing parameters of the sound source object based on the spatial attributes and processing strategy of the sound source object, and generating the rendered bitstream of the speaker channel corresponding to the sound source object based on the co-processing parameters; wherein, the co-processing parameters include at least one of the following: gain parameter, delay parameter, reverberation parameter, and spectrum parameter.

[0027] In this embodiment, custom-designed parameters such as gain, delay, reverberation, and spectrum are generated based on the spatial attributes and processing strategy of the object track. The sound source object is then processed accordingly, enhancing the accurate positioning of the sound source object in three-dimensional space and presenting a more realistic motion trajectory and spatial hierarchy. This makes the sound source object sound more natural and in line with psychological expectations, effectively avoiding sound field chaos. Each sound source object is integrated into the overall environment while remaining clearly distinguishable, achieving the best effect in different dimensions and significantly improving the listener's understanding of the music content and immersive experience.

[0028] In some implementations, generating the multi-channel pseudo-immersive audio stream based on the base sound field bitstream and the object bitstream includes: mixing the base bitstream of each channel in the base sound field bitstream and the rendered bitstream of each channel in the object bitstream to obtain the multi-channel pseudo-immersive audio stream; wherein the multi-channel includes at least three channels selected from the following: left channel, right channel, center channel, subwoofer channel, left surround channel, right surround channel, left rear surround channel, right rear surround channel, front left top channel, front right top channel, rear left top channel, and rear right top channel.

[0029] In this embodiment, the channel signals of the intelligent object stream are mixed with the corresponding channel signals of the base sound field stream to improve the overall clarity and positioning of the audio. This enables flexible compatibility and efficient enhancement of immersive audio, significantly improves the adaptability of multi-speaker playback, sound field integrity and dynamic performance, and is widely compatible with the precise control of object-oriented audio and traditional channels to form a backward-compatible and upward-extending panoramic sound.

[0030] In some implementations, determining the driving signals for multiple target speakers in the current usage scenario based on the acoustic parameters corresponding to the current usage scenario and the pseudo-immersive audio bitstream of the multi-channel system includes: determining the gain coefficient of each of the sound source objects based on the acoustic parameters corresponding to the current usage scenario; for each channel in the multi-channel system, determining the target speaker corresponding to that channel; generating the target bitstream of that channel based on the gain coefficient of the sound source object corresponding to that channel and the bitstream of that channel; and using the target bitstream of that channel as the driving signal for the target speaker corresponding to that channel.

[0031] In this embodiment, firstly, the gain coefficient of each object track is determined based on the acoustic parameters of the usage scenario, significantly improving the clarity, spatial balance, and personalized listening experience of the audio content in a specific environment. Compared to a fixed gain strategy, dynamic adjustment can precisely compensate for acoustic defects in the usage scenario, allowing each sound object to be properly positioned. Secondly, the speakers corresponding to each channel are dynamically configured. Compared to a fixed channel layout, dynamic configuration breaks the limitations of physical placement, allowing the sound to follow the environment and achieving a more intelligent and immersive listening experience. Finally, the bitstream of the corresponding channel is compensated based on the gain coefficient of each sound source object, achieving the goal of combining the flexibility of objectified audio with the stability of the channel system. This ensures that each sound element can "be in its proper place and make its own sound" in three-dimensional space, realizing dynamic sound field balance and precise optimization of listening experience, significantly improving the clarity, spatial layering, and personalized adaptability of audio in complex environments.

[0032] In some implementations, determining the gain coefficient of each sound source object based on the acoustic parameters corresponding to the current usage scenario includes: determining the ideal signal corresponding to each speaker based on the acoustic parameters corresponding to the current usage scenario and the hardware parameters of each speaker; and for each speaker, reconstructing the bitstream of the corresponding channel of the speaker based on the ideal signal of the speaker to obtain the gain coefficient of the sound source object corresponding to the speaker.

[0033] In this embodiment, on the one hand, the reference signal for each speaker is determined based on the acoustic parameters of the usage scenario and the hardware parameters of each speaker. This dual-parameter driven reference signal setting can accurately match the speaker capabilities and compensate for the acoustic defects of the usage scenario, making the sound both accurate and stable. This achieves the physical accuracy of the sound field reproduction and the optimal matching of system performance, significantly improving clarity, dynamic range, and long-term stability. On the other hand, the panoramic sound bitstream is reconstructed based on the ideal signal corresponding to the speaker to generate the gain of the corresponding sound source object. This achieves the purpose of reverse mapping the physical acoustic target to object-level rendering, realizing the WYSIWYG sound field construction, accurately reproducing the sound field and providing personalized listening experience, improving spatial consistency, dynamic response capability, and multi-scene adaptability.

[0034] In some implementations, generating the target bitstream of the channel based on the gain coefficient of the audio source object corresponding to the channel and the bitstream of the channel includes: generating the target bitstream of the channel based on the product between the gain coefficient of the audio source object corresponding to the channel and the bitstream of the channel.

[0035] In the embodiments of this application, gain compensation is used to partially compensate for the loss of sound quality, making the sound clearer and more distinguishable. This can improve the consistency and stability of the listening experience in weak signal environments, achieve an efficient balance between bitstream and listening experience, and effectively improve the user experience.

[0036] This application provides an audio processing system, which includes an audio input module, an extraction module, a mixing module, a reconstruction module, and an audio output module, wherein: The audio input module is used to receive raw audio data; The extraction module is connected to the audio input module and is used to determine key information of the original audio data. The key information includes multiple sound source objects, category information and reverberation information. The category information represents the genre to which the original audio data belongs and / or the emotion of the original audio data. The mixing module, connected to the extraction module, is used to generate a multi-channel pseudo-immersive audio stream based on the multiple sound source objects, the category information, and the reverberation information. The reconstruction module, connected to the mixing module, is used to determine the driving signals of multiple target speakers in the current usage scenario based on the acoustic parameters corresponding to the current usage scenario and the multi-channel pseudo-immersive audio bitstream. The audio output module is connected to the reconstruction module and is used to drive each target speaker to play based on the driving signal of each target speaker.

[0037] In this embodiment, firstly, the extraction module separates different sound source objects, genres, emotions, reverb, etc., from the original audio data, achieving a deep understanding of the audio content and greatly improving the separation quality. This lays the foundation for developing differentiated rendering strategies for audio of different styles, enriching the processing effects and reducing the possibility of unnatural sound fields due to low separation quality. Secondly, the mixing module generates a panoramic sound bitstream based on the sound source objects, genres, emotions, reverb information, etc. In other words, it can upmix two-channel stereo audio to output high-quality multi-dimensional panoramic sound audio with immersion, clear positioning, and content adaptability. This enriches the mixing processing methods, allows for customized adjustments based on the specific characteristics of the audio content, improves the mixed effect, and reduces the possibility of blurred sound fields, false spatial sense, and chaotic instrument positioning caused by rigid or monotonous mixing processing methods, achieving a truly immersive high-quality listening experience. Finally, the reconstruction module determines and outputs the drive signals for each speaker based on the acoustic parameters of the current usage scenario (such as a vehicle, home theater, etc.) and the panoramic sound bitstream. In other words, it can perform precise sound field reconstruction according to the characteristics of the audio and the usage environment, providing listeners with higher quality and more personalized sound effects, thereby greatly improving the listening experience.

[0038] This application also provides a vehicle, including a processor and a memory, wherein the memory stores a computer program or instructions that, when executed by the processor, implement any of the methods described above.

[0039] This application also provides a computer-readable storage medium storing a computer program or instructions that, when executed by a processor, implement any of the methods described above.

[0040] This application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the method described above.

[0041] The beneficial effects of this application are: (1) The training separation model is used to separate each sound source object in the original audio data, which improves the separation efficiency and separation accuracy and significantly reduces the processing cost. At the same time, since the pre-trained separation model has stronger generalization ability and multi-scene adaptability, it can improve the application capability of the method (e.g., simple or complex real-world audio). (2) Converting the original time-domain mixed signal into a complex spectrum can simultaneously preserve amplitude and phase information, achieving more complete and natural speech reconstruction, significantly improving listening quality and processing accuracy. Compared with traditional methods that only process amplitude spectrum, complex spectrum modeling allows the method to understand the spatiotemporal structure of sound, rather than just the volume. (3) The sparse banding strategy is adopted to decompose the speech audio band into multiple sub-bands, and different ratios of downsampling compression are applied to different sub-bands. Banding is performed according to the importance of speech to achieve priority retention of key frequency bands. At the same time, differentiated downsampling compression is used to optimize the computational load and transmission efficiency. This achieves the goal of giving most of the computing power to the low and mid frequencies that are sensitive to the human ear while compressing the high-frequency dimension. This not only reduces the frequency dimension and computational complexity of the model but also has almost no loss of model accuracy. Furthermore, it can extract high-dimensional features suitable for separation tasks, achieving the optimal balance between key information retention and redundant data compression. (4) First, frequency band modeling of high-dimensional features using the frequency domain processing unit can accurately reveal the global periodic structure and frequency fingerprint (i.e., the harmonic distribution and energy concentration area of ​​the signal), enhance the ability to distinguish sound sources, enable the model to clearly hear the essential composition of sound, and significantly improve robustness and efficiency. Second, frame-by-frame modeling of weighted features using the time domain processing unit is used. Since the weighted high-dimensional features have integrated key frequency band information, but the smoothness and dynamic changes of their time axis still need to be finely depicted, frame-by-frame modeling can accurately capture local dynamic changes and temporal continuity, ensuring fluent and natural speech and efficient recognition of instantaneous events. This is a key link in achieving the leap from clear hearing to true hearing. Finally, by using a weighted processing unit to weight the encoded features and the features after frequency band modeling, as well as the features after frequency band modeling and the features after frame number modeling, the goal of dynamically weighting for different times and frequency positions is achieved. This enables differentiated attention to the importance of music features. Specifically, for spectrally rich segments (such as the chorus) or key frequency ranges, the model automatically assigns higher weights to enhance the priority of these regions in the separation process, thereby improving the effectiveness of music source separation. At the same time, it can also capture multi-dimensional dependencies and importance in the input sequence, further improving the model's ability to represent and process complex music data. (5) The sparse banding strategy is adopted to decompose the features after time-frequency domain modeling into multiple sub-bands, and different ratios of upsampling expansion are applied to different sub-bands. This can achieve priority modeling of key information by banding according to importance, and at the same time, differentiated upsampling expansion can optimize the allocation of computing resources, so that the model can allocate attention and computing resources more intelligently under limited computing power, and achieve the synergistic optimization of key frequency band focusing and resource allocation on demand. (6) By using the first analysis model after training to analyze the category, emotion, reverberation information, etc. of the audio, it can achieve end-to-end joint analysis of multi-dimensional acoustic attributes, significantly improving analysis efficiency, consistency and contextual relevance. Compared with step-by-step independent processing, joint modeling avoids error accumulation, reduces information loss and error transmission, and can capture the implicit relationship between category, emotion and acoustic environment. (7) By integrating the object tracks in the audio through the trained second analysis model, the category, emotion, reverberation information, etc. of the audio can be analyzed. It can achieve refined semantic parsing in multi-source scenarios, significantly improve the accuracy of analysis, spatial perception ability and context consistency. Compared with analysis based solely on mixed signals, secondary modeling using separated "object tracks" (such as independent speakers and background sound events) can avoid sound source interference and accurately locate the attribute features of each track. (8) Based on the reverberation information, the virtual acoustic environment in which the audio is most likely located can be deduced, which can realize the reverse modeling of sound field perception, significantly improve the spatial authenticity, rendering adaptability and interactive immersion of audio content. Compared with the environmental assumptions that rely solely on the original waveform or metadata, the reverse inference based on reverberation features can accurately restore the acoustic context and provide a physical basis for virtual reconstruction and intelligent processing. (9) By making fusion decisions on reverberation, category and each sound source object, a unique sound field psychological model is dynamically generated for the audio, realizing the fusion of multi-dimensional perception, constructing an emotional sound field, optimizing cognitive comfort, realizing deep personalization and situational adaptation of listening experience, and significantly improving the emotional resonance, scene immersion and cognitive comfort of audio content. (10) According to the global sound field properties of the sound field effect model, apply matching global environmental reverberation and equalization processing to better simulate the sound propagation characteristics under different environments, form a realistic base sound field code stream, provide a strong and realistic sense of spatial immersion for the entire sound field, thereby improving the realism and immersion of the hearing. (11) Generate exclusive customized parameters such as gain, delay, reverberation, and spectrum based on the spatial attributes and processing strategies of the object track, and process the sound source object accordingly. This enhances the accurate positioning of the sound source object in three-dimensional space and presents a more realistic motion trajectory and spatial hierarchy, making the sound source object sound more natural and in line with psychological expectations. It effectively avoids sound field chaos, so that each sound source object can be integrated into the overall environment while remaining clearly distinguishable, achieving the best effect in different dimensions and significantly improving the listener's understanding of the music content and immersive experience. (12) By mixing the base sound field bitstream and the object bitstream, the overall clarity and positioning of the audio are improved, which can achieve flexible compatibility and efficient enhancement of immersive audio, significantly improve the adaptability of multi-speaker playback, sound field integrity and dynamic performance, and widely support the precise control of objectified audio and traditional channels to form a panoramic sound that is backward compatible and upward extended.

[0042] (13) The reference signal of each speaker is determined according to the acoustic parameters of the usage scenario and the hardware parameters of each speaker. This dual-parameter driven reference signal setting can accurately match the speaker capabilities and compensate for the acoustic defects of the usage scenario, making the sound both accurate and stable, achieving the physical accurate reproduction of the sound field and the optimal matching of system performance, significantly improving clarity, dynamic range and long-term stability. (14) The panoramic sound stream is reconstructed according to the ideal signal corresponding to the loudspeaker to generate the gain of the corresponding sound source object, thereby achieving the purpose of reverse mapping of the physical acoustic target to the object-level rendering, realizing the sound field construction that is exactly what you want, accurately restoring the sound field and personalized listening experience, and improving spatial consistency, dynamic response capability and multi-scene adaptability. (15) Dynamically configure the speakers corresponding to each channel. Compared with the fixed channel layout, dynamic configuration breaks the limitations of physical placement, allowing the sound to follow the environment and achieve a more intelligent and immersive listening experience. (16) The panoramic sound bitstream of the corresponding channel is compensated according to the gain coefficient of each sound source object, which achieves the purpose of combining the flexibility of objectified audio with the stability of the channel system, so that each sound element can "be in its place and make its own sound" in three-dimensional space, realizing the dynamic balance of the sound field and the precise optimization of the listening experience, and significantly improving the clarity, spatial layering and personalized adaptation of audio in complex environments. Attached Figure Description

[0043] Figure 1 A schematic diagram illustrating the implementation flow of an audio processing method provided in an embodiment of this application; Figure 2 A schematic diagram of the composition structure of an audio processing system provided in this application embodiment. Figure 1 ; Figure 3 A schematic diagram of the composition structure of an audio processing system provided in this application embodiment. Figure 2 ; Figure 4 A schematic diagram of a separation model provided in an embodiment of this application; Figure 5 A schematic diagram of the distribution of in-vehicle speakers provided in an embodiment of this application; Figure 6 This is a schematic diagram of a signal processing method provided in an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0044] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0045] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0046] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0047] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0048] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0050] The method provided in this application can be executed by an electronic device, which can be a laptop, tablet, desktop computer, vehicle, set-top box, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0051] The technical solutions in the embodiments of this application will now be clearly and completely described with reference to the accompanying drawings.

[0052] Figure 1 This is a schematic diagram illustrating the implementation flow of an audio processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes steps S11 to S14, wherein: Step S11: Determine the key information of the original audio data. The key information includes multiple sound source objects, category information and reverberation information. The category information represents the genre to which the original audio data belongs and / or the emotion of the original audio data.

[0053] Here, the raw audio data can be in any format, such as mp3, FLAC, WAV, etc. The raw audio data can be acquired using any suitable method, such as online streaming media, Bluetooth, USB (Universal Serial Bus) interface, AUX (Auxiliary Input), etc.

[0054] The raw audio data may include multiple sound source objects (or object tracks). A sound source object refers to a discrete sound entity with independent spatial location, acoustic characteristics, and semantic attributes within an audio scene. It not only represents the physical source of the sound (such as vocals, lead instrument, bass, drums, backing vocals, ambient sounds, etc.), but also carries multi-dimensional information that can be recognized by the auditory system, such as direction, distance, motion trajectory, emotional tone, and cognitive priority. It is understandable that each sound source object is independent of the others and possesses its own type information (such as tags). For example, a single audio file may be separated into multiple independent and distinct sound source objects, such as vocals, lead guitar, bass, and drums.

[0055] Category information may include, but is not limited to, genre information and emotional information. Genre information indicates the genre to which the original audio data belongs, and genres may include, but are not limited to, classical, rock, jazz, electronic, blues, country, disco, hip-hop, metal, pop, reggae, acoustic, etc. Emotional information indicates the emotional characteristics of the original audio data, and these emotional characteristics may include, but are not limited to, rousing, soothing, melancholic, etc.

[0056] Reverberation information can include, but is not limited to, reverberation characteristics, the ratio of direct sound to reverberated sound (or wet / dry ratio), etc. Reverberation characteristics can include, but are not limited to, reverberation time (RT60), early reflection intensity, pre-delay, extension and sound field uniformity, and spectral balance of the reverberation tail. Reverberation characteristics determine the continuity of sound in a space, the sense of space, and the listening quality. Reverberation time (RT60) refers to the time required for the sound pressure level in a room to decrease by 60 decibels after the sound source stops emitting sound; it is a core indicator for measuring the "acoustic activity" of a space. For example, a short RT60 (e.g., 0.6 seconds) is suitable for scenarios with clear speech (such as conference rooms), while a long RT60 (e.g., 1.8 seconds) enhances the fullness of music and is commonly seen in concert halls. Early reflection intensity refers to the sound energy of the first few reflections arriving within 50 milliseconds after the direct sound. Pre-delay is the time interval between the direct sound and the first reflection, reflecting the listener's perceived distance from the reflecting surface. In practice, a larger pre-delay results in a more open sense of space and is often used to create the effect of singing from a distance. The spectral balance of reverberation tails refers to the distribution characteristics of high-frequency, mid-frequency, and low-frequency energy during reverberation decay. Diffusion and sound field uniformity describe the randomness and uniformity of reflected sound distribution in space. The dry-to-wet ratio directly affects the perceived distance and environmental integration of sound. In practice, dry sound often represents closeness and clarity, suitable for dialogue; wet sound often represents immersion and etherealness, suitable for ambient music.

[0057] This key information can be determined in any suitable way.

[0058] In some implementations, the raw audio data can be input into a pre-established extraction model to obtain the key information. This extraction model can be any suitable deep learning neural network model capable of performing this function, such as a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN).

[0059] In some implementations, the raw audio data can be input into a pre-established separation model to obtain multiple audio source objects. This separation model can be any suitable deep learning neural network model capable of achieving this function.

[0060] In some implementations, the raw audio data can be input into a pre-established first analysis model to obtain category information and / or reverberation information. This first analysis model can be any suitable deep learning neural network model capable of performing this function. For example, the first analysis model can be used to analyze the musical features of the raw audio data to identify its genre and reverberation information, and infer its emotional content. Musical features may include, but are not limited to, time-domain features, frequency-domain features, time-frequency features, rhythmic features, tonality features, structural features, and spatial features. Time-domain features may include, but are not limited to, energy envelope, peak level, and transient density; these features are used to determine the dynamic range and rhythmic impact of the audio. Frequency-domain features may include, but are not limited to, spectral centroid, spectral flux, and sub-band energy distribution (low / mid / high frequency ratio); these features are used to identify timbre brightness and frequency congestion. Time-frequency features may include, but are not limited to, MFCC (Mel-Frequency Cepstral Coefficients), chromaticity vector, and harmonic noise ratio; these features are used for timbre recognition and harmonic structure analysis. Rhythmic features may include, but are not limited to, tempo, intensity, regularity, and position of beats. These features are used to determine musical style and rhythm-driven passages. Tonal features may include, but are not limited to, tonality, mode (major / minor), and chord progression outlines. These features are used to determine emotional tone and adapt soundstage width. Structural features may include, but are not limited to, passage boundary detection (intro / verse / chorus / interlude / outro) and passage duration. These features are used to dynamically switch mixing strategies. Spatial features may include, but are not limited to, channel correlation, phase difference, and width estimation of the original stereo sound. These features are used to determine the soundstage type of the original recording (e.g., narrow / wide / phase anomaly).

[0061] In some implementations, the raw audio data and multiple sound source objects can be input into a pre-established second analysis model to obtain category information and / or reverberation information. The second analysis model can be any suitable deep learning neural network model capable of achieving this function.

[0062] Step S12: Generate a multi-channel pseudo-immersive audio stream based on multiple sound source objects, category information, and reverberation information.

[0063] Here, the pseudo-Atmos stream is a stream containing multiple channels, which may include, but are not limited to, left channel (L), right channel (R), center channel (Center), subwoofer channel (LFE), left surround channel (LS), right surround channel (RS), left rear surround channel (LB), right rear surround channel (RB), front left top channel (FLT), front right top channel (FRT), rear left top channel (RLT), and rear right top channel (RRT). For example, this pseudo-Atmos stream may include an 8-channel stream, a 12-channel stream, or a 20-channel stream. It is understood that the specific channels presented by the pseudo-Atmos stream depend on the speakers and audio source (i.e., the original audio data) in the usage scenario.

[0064] The pseudo-panoramic audio stream can be generated in any suitable way.

[0065] In some implementations, the sound source objects, category information, and reverberation information can be input into a first generation model to obtain the pseudo-immersive sound bitstream. The first generation model can be any suitable neural network model capable of performing this function.

[0066] In some implementations, a sound field effect model can be determined first based on reverberation information, category information, and multiple sound source objects. This sound field effect model includes global sound field attributes, spatial attributes of each sound source object, and processing strategies for each sound source object. Then, a base sound field bitstream is generated based on the global sound field attributes and the original audio data. Next, an object bitstream is generated based on the spatial attributes and processing strategies of each sound source object. Finally, the pseudo-immersive sound bitstream is generated based on the base sound field bitstream and the object bitstream.

[0067] Step S13: Based on the acoustic parameters corresponding to the current usage scenario and the multi-channel pseudo-immersive audio bitstream, determine the driving signals of multiple target speakers in the current usage scenario.

[0068] Here, the usage scenario can be any suitable environment where audio technology is applied, with the core function of playing or amplifying sound, relying on a speaker system to achieve audio output. This usage scenario can include, but is not limited to, entertainment scenarios, mobile scenarios, and public sound reinforcement scenarios. Entertainment scenarios can include, but are not limited to, home theaters and cinemas, emphasizing immersive sound effects and high-fidelity reproduction. Mobile scenarios can include, but are not limited to, in-vehicle audio systems (i.e., car stereos), primarily optimizing sound field distribution within a limited space. Public sound reinforcement scenarios can include, but are not limited to, conference halls and theaters, emphasizing sound coverage and clarity.

[0069] The acoustic parameters corresponding to a usage scenario mainly refer to its internal acoustic parameters. It's understandable that different usage scenarios have different acoustic parameters. Acoustic parameters can include, but are not limited to, seating position, interior space dimensions, the sound absorption characteristics of interior materials, and the acoustic performance characteristics of each speaker at each listener's location. Seating position refers to the physical area where the listener sits; its layout directly affects the sound propagation path and the perceived balance of sound. For example, in a vehicle, seating positions are typically divided into front seats (driver's seat, front passenger seat) and rear seats (left, middle, right). Interior space dimensions refer to the length, width, height, and volume of the interior space, determining the sound wave propagation distance and standing wave frequency distribution, and are core factors affecting acoustic modes. The sound absorption characteristics of interior materials are measured by the sound absorption coefficient, reflecting the material's ability to absorb sound energy. The acoustic performance characteristics of each speaker at each listener refer to the sound response (such as sound pressure level, frequency response, phase consistency, direct sound / reflected sound ratio, volume, delay, etc.) of speakers in different positions (such as the front door, rear door, center, surround, etc.) at each listener's ear position, which together determine the sound field positioning, stereo effect and immersion.

[0070] This use case allows for the integration of multiple speakers, such as 12 or 16. These multiple target speakers can be some or all of the speakers, depending on the specific usage requirements. It's understood that "multiple" means at least three.

[0071] The driving signal can be determined in any suitable way.

[0072] In some implementations, the acoustic parameters and the pseudo-panoramic audio bitstream can be input into a pre-established drive determination model to obtain the various drive signals. This drive determination model can be any suitable neural network model capable of performing this function.

[0073] In some implementations, the gain coefficient of each sound source object can be determined first based on acoustic parameters, and then the driving signal can be generated based on the gain coefficient of the sound source object and the bitstream of the channel corresponding to the sound source object.

[0074] Step S14: Distribute the drive signal of each target loudspeaker to the corresponding loudspeaker channel for playback.

[0075] Here, the distribution of drive signals to the loudspeakers is a crucial step in achieving accurate sound field reproduction in an audio system. Its core is to accurately route audio signals to the corresponding physical loudspeaker units according to channel identifiers, ensuring that the spatial positioning of the sound is consistent with the design. Understandably, the drive signals must correspond one-to-one with the physical location of the loudspeakers. In practice, after the distribution is complete, the drive signals can be converted into digital signals for playback.

[0076] In this embodiment, firstly, different sound source objects, genres, emotions, reverb, etc., are separated from the original audio data, achieving a deep understanding of the audio content and greatly improving the separation quality. This lays the foundation for developing differentiated rendering strategies for audio of different styles, enriching the processing effects and reducing the possibility of unnatural sound fields due to low separation quality. Secondly, a panoramic sound bitstream is generated based on the information of each sound source object, genre, emotion, and reverb. In other words, it can upmix two-channel stereo audio to output high-quality multi-dimensional panoramic sound audio with immersion, clear positioning, and content adaptability. This enriches the mixing processing methods, allows for customized adjustments based on the specific characteristics of the audio content, improves the mixed effect, and reduces the possibility of blurred sound fields, false spatial sense, and chaotic instrument positioning caused by rigid or monotonous mixing processing methods, achieving a truly immersive high-quality listening experience. Finally, the driving signals for each speaker are determined and output based on the acoustic parameters of the current usage scenario and the panoramic sound bitstream. In other words, it can accurately reconstruct the sound field according to the characteristics of the audio and the usage environment, providing listeners with higher quality and more personalized sound effects, thereby greatly improving the listening experience.

[0077] In some embodiments, step S11 includes steps S111 and S112, wherein: Step S111: Using a pre-trained separation model, the original audio data is separated to obtain multiple sound source objects.

[0078] Here, the separation model can be any suitable neural network model capable of implementing this function. In some implementations, the separation model may include, but is not limited to, an encoding module, a separation module, and a decoding module. The encoding module is responsible for extracting feature information from the original audio data, the separation module is used for time-frequency domain modeling of the signal, and the decoding module reconstructs the modeled features into a time-domain signal. In some implementations, the original audio data can be input into the separation model to obtain multiple sound source objects. In some implementations, the original audio data can first be converted into a corresponding complex spectrum, and then the complex spectrum can be input into the separation model to obtain the separated complex spectrum. Finally, the separated complex spectrum can be converted into a corresponding time-domain signal (i.e., the sound source object).

[0079] Step S112: Analyze and process the raw audio data to obtain category information and reverberation information.

[0080] Here, category information may include, but is not limited to, genre information, emotional information, etc. Reverb information may include, but is not limited to, reverb characteristics, wet / dry ratio, etc.

[0081] In some implementations, the raw audio data can be input into a first analysis model to obtain category information and reverberation information.

[0082] In some implementations, the original audio data and each sound source object can be input into a second analysis model to obtain category information and reverberation information.

[0083] The analysis model (including the first and second analysis models) can be any suitable neural network model capable of analyzing the genre, emotion, reverberation information, etc., of audio data. It is understood that the second analysis model differs from the first analysis model.

[0084] In some implementations, the analysis model includes a classification model, a sentiment analysis unit, and a reverberation analysis unit. The classification model is used to determine the genre to which the audio data belongs; this model can be any suitable neural network model capable of performing this function. The sentiment analysis unit is used to analyze the emotional characteristics of the audio data; this unit can be any suitable neural network unit capable of performing this function. The reverberation analysis unit is used to determine the reverberation information of the audio data; this unit can be any suitable neural network unit capable of performing this function. In practice, inputting the original audio data into the classification model yields genre information; inputting the original audio data into the sentiment analysis unit yields sentiment information; and inputting the original audio data into the reverberation analysis unit yields reverberation information.

[0085] In this application's implementation, on the one hand, a trained separation model is used to separate the various sound source objects in the original audio data, improving separation efficiency and accuracy while significantly reducing processing costs. Furthermore, since the pre-trained separation model possesses stronger generalization ability and multi-scenario adaptability, it enhances the application capability of this method (e.g., simple or complex real-world audio environments). On the other hand, analyzing and processing the original audio data to obtain reverberation, category, and emotion not only facilitates subsequent inference of the acoustic environment and artistic intent based on reverberation, achieving the goal of realistically recreating the listening experience, but also deeply explores the acoustic and semantic value of sound. This enables precise decision-making and experience upgrades in fields such as music production, intelligent interaction, and psychological care, thereby not only improving the automation level of audio processing but also making human-computer communication more humane.

[0086] In some implementations, the separation model includes an encoding module, a separation module, and a decoding module; step S111 includes steps S1111 to S1114, wherein: Step S1111: Preprocess the original audio data to obtain the complex spectrum corresponding to the original audio data.

[0087] Here, preprocessing mainly involves converting the original time-domain mixed signal into the corresponding complex spectrum. This preprocessing can include, but is not limited to, the Short-time Fourier Transform (STFT).

[0088] The complex spectrum, also known as the frequency domain representation, covers the audible range of the human ear (20Hz–20kHz) and reflects the frequency components within the speech band. The speech band is specifically designed for transmitting clear and intelligible speech information, and its frequency range is 300Hz–3400Hz.

[0089] In some implementations, since the human ear can also perceive phase changes, even if the amplitude is perfectly correct, phase deviations can cause speech to sound "mechanical," "metallic," or unclear. Therefore, complex spectral processing is used to model directly in the complex domain, simultaneously optimizing the real and imaginary parts (i.e., amplitude and phase). By using neural networks to learn how to recover the most likely clean phase, the naturalness and intelligibility of speech are significantly improved.

[0090] In practice, the complex frequency corresponding to the original audio data is used to characterize the frequency range of the speech band of the original audio data.

[0091] Step S1112: Using the encoding module, the complex spectrum is encoded to obtain the first feature.

[0092] Here, the encoding module can be any suitable neural network module capable of implementing this function. The encoding module can be composed of at least two stacked encoder layers. The encoding module can include, but is not limited to, convolutional layers, normalization layers, activation functions (such as GLU (Gated Linear Unit), ReLU, Sigmoid, etc.).

[0093] Encoding processing may include, but is not limited to, at least one of the following: splitting, downsampling, etc. Splitting involves dividing the audio band into at least one sub-band (such as low frequency, mid frequency, high frequency, etc.). Downsampling is a compression process; in practice, the compression ratio (or scale) of different sub-bands can be the same or different. For example, the compression ratio of low frequency can be 1x, while the compression ratio of mid frequency can be 4x, and the compression ratio of high frequency can be 16x.

[0094] The first feature can be a feature obtained by downsampling the entire frequency band, or it can be a feature obtained by merging different sub-bands after downsampling. It is understood that this first feature is a feature of the entire frequency band.

[0095] In some implementations, the complex spectrum is used to characterize the frequency range of the speech band in the original audio data; the step S1112, "encoding the complex spectrum to obtain the first feature", includes: using a preset sparse banding strategy to divide the frequency range of the speech band to obtain multiple first sub-bands; and performing downsampling compression processing on each first sub-band at a corresponding multiple to obtain the first feature.

[0096] Here, different first sub-bands correspond to different frequency ranges, which can be low-frequency, mid-frequency, high-frequency, etc. In implementation, human voice energy distribution is highly concentrated, and different frequency bands contribute significantly to intelligibility and naturalness. For example, the mid-low frequency band (300–2000Hz) carries core speech features such as the fundamental frequency and formants, directly affecting speech recognition and the naturalness of the listening experience; it is a critical area that must be preserved with high fidelity. The high frequency band (>4000Hz) mainly contains voiceless consonant details, which contribute to clarity but have a low signal-to-noise ratio and are easily affected by environmental noise. Therefore, a sparse banding strategy can be adopted to dynamically divide sub-bands according to the semantic importance of frequency bands, achieving priority preservation of key frequency bands. It can be understood that this sparse banding strategy is used to balance the model's computational power and performance.

[0097] Different subbands can correspond to the same or different compression ratios. For example, low-frequency and mid-frequency samples may use the same compression ratio, while mid-frequency and high-frequency samples may use different ratios. Alternatively, low-frequency, mid-frequency, and high-frequency samples may all have different compression ratios. In practice, if all frequency bands are processed uniformly, key speech information will be over-compressed, affecting intelligibility. Furthermore, noise bands will not be effectively suppressed, increasing the burden on subsequent processing. Specifically, if the entire frequency band is compressed at a high ratio, the model resolution will be too low, significantly reducing performance; if the entire frequency band is compressed at a low ratio, although the frequency resolution is high, the lack of effective compression leads to excessive model complexity and computational cost. Therefore, different downsampling compression ratios are used, allowing for slight compression of low-frequency subbands (e.g., retaining a 16kHz sampling rate) to ensure the integrity of the speech backbone; and deep compression or filtering of high-frequency subbands (e.g., reducing to 8kHz) to reduce bandwidth requirements and inference latency. In some implementations, a learnable compression gating mechanism can be used, allowing the model to automatically determine the compression ratio for each subband.

[0098] During implementation, by incorporating the downsampling depth into the model, end-to-end joint optimization is supported, reducing the accumulation of information loss.

[0099] In this way, the sparse banding strategy is used to decompose the audio band into multiple sub-bands, and different sub-bands are downsampled and compressed at different ratios. Banding is based on the importance of speech to prioritize the preservation of key frequency bands. At the same time, differentiated downsampling compression optimizes the computational load and transmission efficiency. This achieves the goal of allocating most of the computing power to the low and mid frequencies that are sensitive to the human ear while compressing the high-frequency dimension. This not only reduces the frequency dimension and computational complexity of the model with almost no loss of model accuracy, but also extracts high-dimensional features suitable for separation tasks, achieving the optimal balance between preserving key information and compressing redundant data.

[0100] During implementation, the complex spectrum is input into the encoding module, which allows the input of the first feature.

[0101] Step S1113: Using the separation module, perform time-frequency domain modeling on the first feature to obtain the second feature.

[0102] Here, the separation module can be any suitable neural network module capable of implementing this function. This separation module can be, but is not limited to, at least one of a frequency domain processing unit, a weighted processing unit, and a time domain processing unit. The frequency domain processing unit is used to model the frequency bands of the features. This frequency domain processing unit can be any suitable neural network unit capable of implementing this function, and can include, but is not limited to, gated attention units (GAU), bidirectional long short-term memory networks (BLSTM), etc. The weighted processing unit is used to perform weighted processing on different features. This weighted processing unit can be any suitable neural network unit capable of implementing this function; for example, this weighted processing unit includes attention mechanisms, multi-head attention, etc. The time domain processing unit is used to model the frame count of the features. This time domain processing unit can be any suitable neural network unit capable of implementing this function, and can include, but is not limited to, GAU, BLSTM, etc.

[0103] In some implementations, the separation module includes a frequency domain processing unit and a time domain processing unit.

[0104] In some implementations, the separation module includes a frequency domain processing unit, a weighting processing unit, and a time domain processing unit, wherein the number of weighting processing units can be one, two, or more.

[0105] The second feature can be a feature derived from time-frequency domain modeling across the entire frequency band. It can be understood that this second feature is a full-frequency band feature. In implementation, the first feature is input into the separation module, allowing the input of the high-dimensional second feature.

[0106] Step S1114: Using the decoding module, the second feature is decoded to obtain multiple sound source objects.

[0107] Here, the decoding module can be any suitable neural network module capable of implementing this function. The decoding module can consist of at least two stacked decoder layers. The decoding module may include, but is not limited to, transposed convolutional layers, normalization layers, activation functions, etc.

[0108] Decoding processing may include, but is not limited to, at least one of the following: splitting, upsampling, etc. Splitting involves dividing the audio band into at least one sub-band (e.g., low-frequency, mid-frequency, high-frequency, etc.). Upsampling involves expansion processing; in practice, the expansion ratio (or proportion) of different sub-bands may be the same or different. For example, the expansion ratio of low frequencies may be 1x, while the expansion ratio of mid frequencies may be 4x, and the expansion ratio of high frequencies may be 16x. In some implementations, the expansion ratio of the sub-bands corresponds to the compression ratio.

[0109] In some implementations, step S1114 includes: dividing the frequency range of the second feature using a preset sparse banding strategy to obtain multiple second sub-bands; and performing upsampling expansion processing on each second sub-band at a corresponding multiple to obtain multiple sound source objects.

[0110] Here, different second sub-bands correspond to different frequency ranges, such as low frequency, mid frequency, and high frequency. In implementation, the energy distribution of speech signals exhibits significant non-uniformity. Therefore, a sparse banding strategy can be adopted to dynamically divide sub-bands based on the semantic importance of frequency bands, enabling priority modeling of key information. It can be understood that this sparse banding strategy is used to balance the computational power and performance of the model.

[0111] Different subbands can correspond to the same or different expansion rates. For example, low-frequency and mid-frequency samples may use the same expansion rate, while mid-frequency and high-frequency samples may use different expansion rates. Alternatively, low-frequency, mid-frequency, and high-frequency samples may all have different expansion rates. In implementation, if all bands use the same expansion rate, key low-frequency information may not be fully restored, and high-frequency noise may be excessively amplified. Therefore, different upsampling expansion rates are used, allowing high-ratio upsampling (e.g., ×4) for low-frequency subbands to finely restore intonation fluctuations and formant structures, improving speech naturalness; and low-ratio upsampling (e.g., ×1 or ×2) for high-frequency subbands to suppress noise reconstruction and avoid "hissing" distortion. In some implementations, a learnable expansion gating mechanism can be combined, allowing the model to automatically determine the required expansion rate for each subband. It is understood that upsampling is the inverse process of downsampling in the aforementioned coding module; therefore, the upsampling and downsampling rates must be matched. For example, if the high frequencies in the encoding module are downsampled by 16 times, then the high frequencies in the decoding module need to be upsampled by 16 times; if the intermediate frequencies in the encoding module are downsampled by 4 times, then the intermediate frequencies in the decoding module need to be upsampled by 4 times; if the low frequencies in the encoding module are downsampled by 1 time, then the low frequencies in the decoding module need to be upsampled by 1 time.

[0112] In implementation, by incorporating the upsampling depth into the model, end-to-end joint optimization is supported, reducing the accumulation of information loss.

[0113] In this way, by adopting a sparse banding strategy to decompose the features after time-frequency domain modeling into multiple sub-bands and performing upsampling expansion processing on different sub-bands at different ratios, it is possible to divide the bands according to importance to achieve priority modeling of key information. At the same time, differentiated upsampling expansion optimizes the allocation of computing resources, enabling the model to allocate attention and computing resources more intelligently under limited computing power, and achieving synergistic optimization of key frequency band focusing and on-demand resource allocation.

[0114] It is understandable that the decoding module can be viewed as the inverse process of the encoding module. Upsampling restores the high-dimensional features modeled by the separation module to the complex spectrum form of the separated signal. In implementation, inputting the second feature into the decoding module allows it to output the high-dimensional complex spectrum of the separated signal.

[0115] In some implementations, the complex spectrum can be subjected to an inverse short-time Fourier transform (ISTFT) to obtain the time-domain sampled signals of each separated sound source object.

[0116] In this embodiment, firstly, the original time-domain mixed signal is converted into a complex spectrum, which simultaneously preserves amplitude and phase information, achieving a more complete and natural speech reconstruction and significantly improving listening quality and processing accuracy. Compared to traditional methods that only process the amplitude spectrum, complex spectrum modeling allows the method to understand the spatiotemporal structure of sound, not just volume. Secondly, the high-dimensional features of the complex spectrum are extracted through an encoding module. This not only deeply integrates the joint representation of amplitude and phase, fully preserving the complete physical meaning of the waveform and significantly improving the accuracy and naturalness of the task, but also uncovers high-dimensional hidden patterns (such as capturing the coherence relationship between different frequencies and the phase evolution law between time and frequency), enhancing the modeling ability for complex acoustic scenes. Compared to methods that only process the amplitude spectrum or low-dimensional features, this approach fully releases the expressive potential of complex domain data, enabling the model to better distinguish overlapping speech and separate reverberation components. Furthermore, by performing time-frequency domain modeling on high-dimensional features through a separation module, the complex signal is decomposed into two collaborative but independent processing paths. This fully leverages the complementary advantages of the time and frequency domains, achieving joint modeling of "local dynamics" and "global structure," significantly improving accuracy, robustness, and interpretability. Finally, the decoding module reconstructs the modeled features into a time-domain signal, enabling high-fidelity, low-distortion speech reconstruction. This significantly improves the naturalness of the sound and the practicality of the task, ensuring that the reconstructed speech is closer to real human voices in terms of detail, rhythm, and phase.

[0117] In some implementations, the separation module includes a frequency domain processing unit, a weighted processing unit, and a time domain processing unit; step S1113 includes: using the frequency domain processing unit to perform frequency band modeling on the first feature to obtain a third feature; using the weighted processing unit to perform weighted processing on the first feature and the third feature to obtain a fourth feature; using the time domain processing unit to perform frame number modeling on the fourth feature to obtain a fifth feature; and using the weighted processing unit to perform weighted processing on the fourth feature and the fifth feature to obtain a second feature.

[0118] Here, the frequency domain processing unit may include, but is not limited to, GAU, BLSTM, etc. In some implementations, the frequency domain processing unit may be composed of stacked GAU and BLSTM. The third feature is the high-dimensional feature obtained after frequency band modeling of the encoded complex spectrum. In implementation, since audio objects such as human voices, musical instruments, and noise have unique "frequency fingerprints" in the frequency domain, such as human voice energy concentrated in 300–3400Hz and having obvious fundamental frequency and harmonic structure; environmental noise (such as air conditioner hum) is mostly concentrated in the low frequency band, while keyboard sounds appear as high-frequency random pulses; musical accompaniment often covers a wide frequency band, but the rhythmic components present periodic spectral lines in the frequency domain, the frequency domain processing unit can analyze the high-dimensional feature by frequency band slice, model its statistical characteristics band by band, and thus more accurately distinguish the target speech from the interference source. In implementation, the first feature can be input into the frequency domain processing unit to obtain the third feature.

[0119] Weighted processing units can include, but are not limited to, attention mechanisms and multi-head attention mechanisms. Attention mechanisms extract local and global information from audio signals using convolutional or fully connected layers, then calculate the correlation matrix by evaluating the correlation between different frequencies / frames. Larger values ​​in the correlation matrix indicate a higher correlation between elements, which receive greater weight during model training. Multi-head attention mechanisms can be understood as a collection of multiple attention mechanisms, using different dimensions to obtain different attention mechanisms and correlation matrices. These matrices are then concatenated and fused to obtain the final correlation weight matrix. The ultimate goal is to inform the model which frequencies and frames are more important, thus assigning them more weight (attention). In implementation, the first and third features are input into the weighted processing unit, which then outputs the fourth feature.

[0120] The temporal processing unit may include, but is not limited to, GAU, BLSTM, etc. In some implementations, the temporal processing unit may be composed of stacked GAU and BLSTM. The fourth feature is a high-dimensional feature obtained by modeling the weighted fourth feature with a number of frames. In implementation, since the weighted high-dimensional feature has already incorporated key frequency band information, but its smoothness and dynamic changes on the time axis still need to be finely depicted, the temporal processing unit operates directly on the time series, which can learn the transition rules between frames and avoid unnatural phenomena such as "stuttering" and "jumping" in the reconstructed speech; it preserves prosodic features such as speech rate, pauses, and stress, improving intelligibility and emotional expression; it corrects the time alignment deviation caused by sparse sampling in the frequency domain, ensuring that lip movement and speech are synchronized. At the same time, the temporal processing unit has high temporal resolution, which can accurately locate and model these transient components, enhancing robustness to sudden interference. Furthermore, since the modeling errors of weighted features may accumulate over time, leading to overall distortion, a temporal processing unit is introduced to recover the original temporal structure from deep within the encoder and actively correct deviations in preceding modules, achieving joint error compensation. In implementation, the fourth feature is input into the temporal processing unit to obtain the fifth feature.

[0121] In some implementations, a dual-path structure (i.e., dual GAU and BLSTM) is used. In the frequency domain dimension F, BLSTM is used, so that the output of the current frequency point is determined not only by the information of the current frequency point but also by the information of the upper and lower frequency points. In the time domain dimension T, causal GAU and unidirectional LSTM are used, so that the output of the current frame number is determined not only by the information of the current frame but also by the information of past frames, and is independent of historical frames.

[0122] In some implementations, since each instrument has its own frequency band characteristics, such as bass mainly concentrated in the mid-low frequency range, human voice energy mainly concentrated in the low frequency range, and drum energy concentrated in the entire frequency band and changing rapidly, the model will learn autonomously based on these differences in frequency band distribution and assign different weights and attention to different audio tracks (i.e., sound source objects) in the frequency band to achieve better separation results.

[0123] The second feature can be a high-dimensional feature output by weighting the fourth and fifth features.

[0124] In this embodiment, firstly, frequency domain processing units are used to perform frequency band modeling on high-dimensional features, which can accurately reveal the global periodic structure and frequency fingerprint (i.e., the harmonic distribution and energy concentration areas of the signal), enhance the ability to distinguish sound sources, enable the model to clearly hear the essential composition of sound, and significantly improve robustness and efficiency. Secondly, frame-by-frame modeling is performed on the weighted features through time domain processing units. Since the weighted high-dimensional features have already incorporated key frequency band information, but the smoothness and dynamic changes of their time axis still need to be finely depicted, frame-by-frame modeling can accurately capture local dynamic changes and temporal continuity, ensuring fluent and natural speech and efficient recognition of instantaneous events. This is a key step in achieving the leap from clear hearing to true hearing. Finally, by using a weighted processing unit to weight the encoded features and the features after frequency band modeling, as well as the features after frequency band modeling and the features after frame number modeling, the goal of dynamically weighting for different times and frequency positions is achieved. This enables differentiated attention to the importance of musical features. Specifically, for spectrally rich segments (such as the chorus) or key frequency ranges, the model automatically assigns higher weights to enhance the priority of these regions in the separation process, thereby improving the effectiveness of music source separation. At the same time, it can also capture multi-dimensional dependencies and importance in the input sequence, further improving the model's ability to represent and process complex music data.

[0125] In some embodiments, the method further includes steps S101 to S103, wherein: Step S101: Input the training samples into the separation model to be trained to obtain the predicted multiple sound source objects; Step S102: Determine the target loss value based on the label information corresponding to the predicted multiple sound source objects and training samples; Step S103: Based on the target loss value, update the model parameters of the separation model at least once to obtain the trained separation model.

[0126] Here, before using the separation model, it needs to be trained first. In some implementations, a database (or training set) can be used first, and a corresponding loss function can be set to optimize learning efficiency, avoid overfitting, and improve separation accuracy. The database includes at least one training sample, which is understood to be audio data. The training sample has corresponding label information, which is used to indicate the various sound source objects contained in the training sample. In implementation, through a supervised pre-training phase, the model directly uses the label information to optimize parameters, avoiding the indirect learning path of "clustering first and then aligning" in unsupervised methods, thereby accelerating model convergence and shortening the training cycle. At the same time, the label information guides the model to focus on the feature dimensions that are sensitive to classification, suppressing irrelevant mutations. In the learned representation space, samples of the same class are more compact, and samples of different classes are more separated.

[0127] The training set can be determined in any suitable way. In some implementations, an open-source database can be used as the training set. In other implementations, some audio data from mixing and production can be supplemented to the open-source database.

[0128] In practice, the training samples are input into the separation model, which can then separate multiple sound source objects from the training samples.

[0129] The loss function employs a composite loss function that integrates time-frequency domain information, considers both amplitude and phase characteristics, and incorporates an adaptive weighting mechanism to more comprehensively measure the difference between the estimated signal and the target signal, thereby improving model performance. This target loss value can be, but is not limited to, at least one of the following: frequency domain complex spectrum loss, SDR (Signal-to-Distortion Ratio) normalized loss, and time domain compression loss. For example, the target loss value can be obtained by weighting the frequency domain complex spectrum loss, SDR normalized loss, and time domain compression loss.

[0130] The model parameters can be updated using at least one of the following methods, including but not limited to gradient descent, momentum update, and Newton's momentum method. In implementation, those skilled in the art can determine the update method according to actual needs; this application does not impose any limitations.

[0131] In some implementations, based on preset conditions, it can be determined whether the model parameters of the separation model need to be updated. If updating the model parameters is necessary, a suitable parameter update algorithm is used to update the model parameters. After updating the model parameters, multiple new sound source objects are re-predicted based on the next training sample. Based on the new multiple sound source objects and the label information corresponding to the next training sample, a new target loss value is determined, and it is determined whether the model parameters of the separation model need to be further updated based on the new target loss value. If updating the model parameters is not necessary, the finally updated separation model is determined as the trained separation model. The preset conditions may include, but are not limited to, at least one of the following: number of iterations, target loss value not exceeding a threshold, and target loss value convergence. For example, the target loss value is compared with a threshold; if the target loss value is greater than the threshold, the model parameters of the separation model are updated; if the target loss value is not greater than the threshold, the separation model is determined as the trained separation model. For example, the target loss value is compared with the previous target loss value. If the target loss value is greater than the previous target loss value, the model parameters of the separation model are updated; if the target loss value is approximately equal to the previous target loss value, the separation model is determined as the trained separation model.

[0132] In some implementations, the number of training rounds (e.g., 200 rounds) can be set according to the size of the training set and the model's convergence.

[0133] In some implementations, a suitable optimizer (such as Adam) can be selected and a reasonable initial learning rate (such as 0.0005) can be set, with a learning rate decay strategy employed as training progresses.

[0134] In some implementations, the batch size for training the model can be any suitable size, such as 32, 48, 16, etc.

[0135] In this embodiment, training the separation model using training samples and their label information significantly improves the model's convergence speed, separation accuracy, and generalization ability, especially in scenarios with small sample sizes, effectively mitigating the risk of overfitting. Furthermore, compared to unsupervised pre-training, supervised training allows the model to learn task-related discriminative features in the initial stage, laying a solid foundation for subsequent fine-tuning.

[0136] In some embodiments, step S112 includes step S1121 or step S1122, wherein: Step S1121: Input the raw audio data into the pre-trained first analysis model to obtain category information and reverberation information; Step S1122: Input the original audio data and multiple sound source objects into the pre-trained second analysis model to obtain category information and reverberation information.

[0137] Here, the analysis model integrates audio classification, emotion recognition, reverberation estimation, etc., and realizes end-to-end joint reasoning. At the same time, the joint multi-task enables the model to learn a general speech representation space, enhancing its cross-task generalization ability.

[0138] Category information may include genre information and / or emotional information. Reverb information may include, but is not limited to, reverb characteristics and / or wet / dry ratio.

[0139] In some implementations, the raw audio data can be input into a first analysis model to obtain category information and reverberation information.

[0140] In some implementations, traditional models struggle to distinguish the contributions of different sound sources when applied to mixed audio, leading to confusion in emotion assessment (e.g., misjudging background arguments as the main speaker's emotion), ambiguous category labels (e.g., failing to identify "alarm" sounds mixed into "meeting"), and biased reverberation estimation (distorted modeling due to different spatial locations of sound sources). Therefore, by introducing individual sound source objects and analyzing each independent track (e.g., speaker A, ambient music, sudden noise), the objects and emotions are precisely linked. Simultaneously, each object track carries independent acoustic cues (e.g., direction of arrival, early reflections, attenuation characteristics), providing a more reliable reverberation time (RT60) estimate for the second analysis model. This allows for the inference of the subspace where the sound source is located; it distinguishes between near-field and far-field sound sources, aiding in determining speaker distance and interaction intent; and it dynamically focuses on high-confidence tracks to suppress residual crosstalk.

[0141] Before using the analysis model, it needs to be trained. In some implementations, training samples can be input into the analysis model to be trained to obtain predicted class information and predicted reverberation information. Then, based on the predicted class information, reverberation information, and label information corresponding to the training samples, a target loss value is determined. Finally, the model parameters of the analysis model are updated at least once based on the target loss value to obtain the trained analysis model. In practice, the training process of the analysis model can refer to the training process of the aforementioned separation model.

[0142] In some implementations, the analysis model may include a classification model, which may include a CNN. In some implementations, the CNN in the classification model may be composed of stacked 2D convolutional layers, max pooling layers, fully connected layers, dropout layers, and activation functions. The 2D convolutional layers are used for feature extraction, the max pooling layers for downsampling, the fully connected layers for capturing global information, dropout is used to prevent overfitting, and the final output layer may use a softmax activation function for classification. In some implementations, the classification model may use Mel spectra as input. Since the Mel spectra of different music genres have some unique differences, this model cleverly transforms the music genre classification problem into an image classification task.

[0143] Before using a classification model, it needs to be trained. In some implementations, training samples can be input into the classification model to be trained to obtain predicted faction information. Then, based on the predicted faction information and the label information corresponding to the training samples, a target loss value is determined. Finally, the model parameters of the classification model are updated at least once based on the target loss value to obtain the trained classification model. In practice, the training process of the classification model can refer to the training process of the aforementioned separation model.

[0144] In some implementations, the target loss value of the classification model can be the classification cross-entropy function.

[0145] In some implementations, the training samples for this classification model can come from open-source datasets (such as GTZAN).

[0146] In some implementations, the number of training rounds (e.g., 80 rounds) of the classification model can be set according to the size of the training set and the model's convergence.

[0147] In some implementations, a suitable optimizer can be selected and a reasonable initial learning rate (e.g., 0.001) can be set, with a learning rate decay strategy employed as training progresses.

[0148] In some implementations, the batch size for training the classification model can be any suitable size, such as 32, 48, 16, etc.

[0149] In some implementations, the analysis model includes an analysis unit for analyzing the emotional features and / or reverberation information of the audio data. This analysis unit can be any suitable neural network unit capable of performing this function. For example, the analysis unit may include an emotion analysis unit and a reverberation analysis unit.

[0150] In the embodiments of this application, on the one hand, by using a trained analysis model to analyze the category, emotion, reverberation information, etc. of the audio, end-to-end joint analysis of multi-dimensional acoustic attributes can be achieved, significantly improving analysis efficiency, consistency, and contextual relevance. Compared with step-by-step independent processing, joint modeling avoids error accumulation, reduces information loss and error propagation, and can capture the implicit relationship between category, emotion, and acoustic environment. On the other hand, by using a trained second analysis model to integrate the object tracks within the audio to analyze the category, emotion, reverberation information, etc. of the audio, refined semantic analysis in multi-sound source scenarios can be achieved, significantly improving the accuracy of analysis, spatial awareness, and contextual consistency. Compared with analysis based solely on mixed signals, secondary modeling using separated "object tracks" (such as independent speakers and background sound events) can avoid sound source interference and accurately locate the attribute characteristics of each track.

[0151] In some embodiments, step S12 includes steps S121 to S124, wherein: Step S121: Based on reverberation information, category information, and multiple sound source objects, determine the sound field effect model; wherein, the sound field effect model includes global sound field attributes, spatial attributes of each sound source object, and processing strategy for each sound source object.

[0152] Here, global sound field properties can be used to define the overall spatial size (or perceived spatial size) and environmental reverberation characteristics of a virtual acoustic environment. These environmental reverberation characteristics are related to the size of the space, wall materials, and other factors. Perceived spatial size refers to the perceived size of the environment in which the audio data is located, which can usually be expressed qualitatively or quantitatively as huge, medium, or small.

[0153] The spatial attributes of a sound source object are used to specify a target spatial coordinate (e.g., center, behind the listener) and diffusion (e.g., diffusion according to points, lines, or surfaces). For example, the person is in the center, and the drum is behind the listener.

[0154] The processing strategy specifies the required audio processing type or method (such as volume, reverb, elevation angle, diffusion angle, etc.) and intensity range (such as volume being higher when closer and lower when farther away) for each audio source object.

[0155] The sound field effect model can be determined in any suitable way.

[0156] In some implementations, a correspondence between each reverberation information, each category information, each sound source object, and each sound field effect model can be established in advance. Based on this correspondence, a sound field effect model that is compatible with the reverberation information, category information, and multiple sound source objects can be obtained.

[0157] In some implementations, reverberation information, category information, and multiple sound source objects can be input into the effect determination model to obtain the sound field effect model. This effect determination model can be any suitable model capable of achieving this function.

[0158] In some implementations, step S121 includes: determining the virtual acoustic environment corresponding to the original audio data based on reverberation information; and determining a sound field effect model from a preset rule base based on category information, multiple sound source objects, and the virtual acoustic environment.

[0159] Here, reverberation information may include, but is not limited to, reverberation characteristics, the ratio of direct sound to reverberated sound, etc.

[0160] A virtual acoustic environment is a digital space that simulates the acoustic characteristics of the real world using computer technology. It can reproduce the physical behaviors of sound propagation, reflection, absorption, and reverberation in three-dimensional space. This virtual acoustic environment can include, but is not limited to, concert halls, recording studios, livehouses, and cathedrals. For example, if the original audio data has a long reverberation time, rich early reflections, and slow low-frequency energy decay, it indicates that the original audio data was likely recorded in a concert hall; if the original audio data has a short reverberation time and retains more high-frequency components, it indicates that the original audio data was likely recorded in a recording studio or a small room. In implementation, since reverberation is the "fingerprint" of a room's acoustic characteristics, its decay curve (such as RT60), early reflection distribution, energy diffusion mode, and other parameters can be used to infer room dimensions (volume and surface area), surface material sound absorption coefficients (such as carpet, concrete, and glass), and the relative positions of the sound source and the listener. Therefore, by accurately reproducing spatial geometry and material characteristics through reverberation features, the realism of sound field modeling is enhanced, achieving consistency between soundscape and visual perception.

[0161] The virtual acoustic environment can be determined in any suitable way. In some implementations, a correspondence between each reverberation information and each virtual acoustic environment can be pre-established. Based on this correspondence, a virtual acoustic environment adapted to the reverberation information can be obtained. For example, the virtual acoustic environment can be inferred based on the RT60 duration. For instance, 0.2–0.4s could represent a recording studio or broadcasting studio; 0.4–0.6s could represent a home theater or conference room; 0.6–1.5s could represent a small concert hall; and 1.5s–4.0s+ could represent a cathedral or large auditorium. In some implementations, the reverberation information can be input into a pre-established acoustic environment analysis model to obtain the virtual acoustic environment corresponding to the original audio data. This acoustic environment analysis model can be any suitable neural network model capable of achieving this function.

[0162] This rule base can be a structured knowledge base or rule set that stores optimal sound field construction rules and mixing strategies set by professional audio engineers for different combinations of music attributes (such as emotion + virtual acoustic environment + genre). In other words, the rule base stores different sound field effect models. During implementation, the sound field effect model that is suitable for the category information, each sound source object, and the virtual acoustic environment is determined from multiple sound field effect models.

[0163] In some implementations, the rule base may be a mapping relationship storing "IF (condition) THEN (action)". For example: IF (genre = "symphony" AND emotion = "grand" AND virtual environment = "concert hall") THEN (generate a sound field effect model, where the parameters are: violins are located 30° to the left front, with low diffusion; cellos are located 20° to the right front, with low diffusion; brass instruments are located at the rear, with medium diffusion; reverberation type is Hall; overall reverberation time = 2.0 seconds...).

[0164] In this way, on the one hand, by inferring the most likely virtual acoustic environment of the audio based on reverberation information, inverse modeling of sound field perception can be achieved, significantly improving the spatial realism, rendering adaptability, and interactive immersion of audio content. Compared to environmental assumptions based solely on the original waveform or metadata, inverse inference based on reverberation characteristics can accurately restore the acoustic context, providing a physical basis for virtual reconstruction and intelligent processing. On the other hand, dynamically selecting appropriate sound field effect models based on emotion, genre, object tracks, and virtual acoustic environment can achieve a deep integration of personalized, contextualized, and immersive acoustic experiences, significantly improving the emotional expressiveness, style fidelity, and spatial realism of audio content. Compared to static and uniform sound field processing, dynamic adaptation strategies allow sound to truly change according to the person, the emotion, and the scene.

[0165] Step S122: Generate the base sound field bitstream based on the global sound field attributes and the original audio data.

[0166] Here, the base sound field bitstream is the background sound field generated after applying specific processing to the original audio data. It is used to create an overall sense of space, providing a sense of immersion and environmental atmosphere for the entire three-dimensional sound field. Specific processing may include, but is not limited to, reverb processing and equalization processing. Reverb processing is a technique that adds a sense of space and atmosphere to audio by simulating the reflection and attenuation of sound in physical space. Its core is to restore the spatial characteristics of sound in a real environment, allowing listeners to perceive the "place" where the sound is located. Equalization processing is an audio processing method that optimizes the clarity and auditory structure of sound by adjusting the volume balance of different frequency bands. Its core is to gain or attenuate specific frequency ranges in the audio signal to improve timbre balance, highlight key elements, or eliminate interference.

[0167] In some implementations, the original audio data can be copied at least twice as a sound field basis signal, and the sound field basis signal can be subjected to specific processing to generate a basis sound field bitstream. For example, the sound field basis signal can be reverberated to generate a basis sound field bitstream. Another example is that the sound field basis signal can be equalized to generate a basis sound field bitstream. Yet another example is that the sound field basis signal can first be reverberated, and then the reverberated signal can be equalized to generate a basis sound field bitstream.

[0168] In some implementations, step S122 includes: determining a reverberation processing strategy and an equalization processing strategy based on global sound field properties; performing reverberation processing on the original audio data according to the reverberation processing strategy to obtain first audio data; and performing equalization processing on the first audio data according to the equalization processing strategy to obtain a base sound field bitstream.

[0169] Here, reverb processing strategy (or reverb processing principle) is used to indicate the specific method of reverb processing. In practice, reverb processing is not simply "adding reverb," ​​but rather a purposeful design of the auditory spatial position and emotional color of the sound, ensuring that it blends into the overall mix without compromising clarity. This reverb processing strategy can include, but is not limited to, matching reverb according to the sound source type (e.g., using plate reverb to highlight brightness for lead vocals, using hall reverb to create a sense of grandeur for harmonies, and using room reverb to enhance realistic reflections for the drum kit), refining control parameters (e.g., pre-delay (20–50ms) to ensure clarity of the dry sound, high-pass / low-pass filtering to purify the reverb frequency band, and matching decay time to the rhythm of the song), and using the send channel (i.e., maintaining dry and wet separation to facilitate unified spatial processing of multiple tracks and avoid timbre distortion), etc.

[0170] In some implementations, a target convolutional kernel can be determined from multiple convolutional kernels based on the virtual acoustic environment and global sound field properties (i.e., the overall spatial size). The convolutional kernel (or impulse response) characterizes the reverberation processing strategy corresponding to the virtual acoustic environment at a specific spatial size. In practice, convolving the raw audio data with the impulse response of a specific space simulates the acoustic characteristics of that space. In some implementations, there is a close correlation between the virtual acoustic environment and the perceived spatial size. For example, when the virtual acoustic environment is a church, the perceived spatial size is often large, thus requiring the selection of a convolutional kernel with a longer reverberation time and a wider spatial sense.

[0171] Equalization processing strategies (or equalization processing principles) are used to indicate the specific methods of equalization processing. These equalization processing strategies may include, but are not limited to, prioritizing subtraction over addition (e.g., first removing interfering frequency bands (such as low-frequency hum in vocals, formants in instruments), and then appropriately enhancing the required details), and functional management of frequency bands (e.g., controlling the rhythmic foundation of low frequencies to avoid muddiness; mid-frequencies affecting the intelligibility of vocals and instruments, requiring careful avoidance; increasing the sense of airiness and presence in high frequencies, but preventing harshness), etc.

[0172] Different global sound field properties can correspond to the same or different reverberation processing strategies and equalization processing strategies.

[0173] In some implementations, a correspondence between each global sound field attribute and each reverberation processing strategy can be pre-established. Based on this correspondence, a reverberation processing strategy adapted to the global sound field attributes can be obtained. For example, churches / concert halls have long reverberation times and many reflections.

[0174] In some implementations, a correspondence between each global sound field attribute and each equalization processing strategy can be established in advance. Based on this correspondence, an equalization processing strategy adapted to the global sound field attributes can be obtained.

[0175] In some implementations, the correspondence between each global sound field attribute, each reverberation processing strategy, and each equalization processing strategy can be established in advance. Based on this correspondence, a reverberation processing strategy and an equalization processing strategy adapted to the global sound field attributes can be obtained.

[0176] During implementation, equalization processing of the first audio data after reverberation is beneficial for further optimizing the audio data after convolutional reverberation processing, making the audio data sound more natural and comfortable in a specific spatial environment.

[0177] In this way, by applying matching global environmental reverberation and equalization processing based on the global sound field properties of the sound field effect model, the sound propagation characteristics under different environments can be better simulated, forming a realistic base sound field bitstream, providing a strong and realistic sense of spatial immersion for the entire sound field, thereby improving the realism and immersion of the hearing.

[0178] Step S123: Generate an object bitstream based on the spatial attributes of each sound source object and the processing strategy of each sound source object.

[0179] Here, different sound source objects can have different spatial attributes, which are used to represent the location information of the sound source object in the three-dimensional virtual sound field. The processing strategies for different sound source objects can be the same or different.

[0180] The object bitstream can include the rendered bitstream of the speaker channel corresponding to each audio source object. A speaker channel refers to a physical or logical channel used to play sound in a multi-channel audio system. Different speaker channels correspond to different physical speaker positions (e.g., front left, front right, center, surround left, surround right, subwoofer, etc.).

[0181] The object's bitstream can be generated in any suitable way.

[0182] In some implementations, the spatial attributes and processing strategies of each audio source object can be input into a second generative model to obtain the object's bitstream. This second generative model can be any suitable neural network model capable of performing this function.

[0183] In some implementations, step S123 includes: for each sound source object, determining the collaborative processing parameters of the sound source object based on the spatial attributes and processing strategy of the sound source object, and generating the rendering bitstream of the speaker channel corresponding to the sound source object based on the collaborative processing parameters; wherein the collaborative processing parameters include at least one of the following: gain parameter, delay parameter, reverberation parameter, and spectrum parameter.

[0184] Here, the co-processing parameters can be a structured dataset containing a series of specific parameters used to perform the perceptual optimization processing chain. These co-processing parameters ensure that the final rendered sound has a natural sense of space and an immersive experience.

[0185] Gain parameter refers to the amplification or attenuation ratio of the audio source signal by each speaker channel in multi-channel rendering.

[0186] The gain parameter can be determined in any suitable way.

[0187] In some implementations, different sound source objects can correspond to the same or different gain parameters. For example, a correspondence between each sound source object and each gain parameter can be established in advance, and the gain parameter can be obtained based on this correspondence.

[0188] In some implementations, the gain parameter can be determined based on the target spatial location of the sound source object and the first intensity range corresponding to the sound source object, thereby ensuring a reasonable volume distribution of the sound source object among different speakers to achieve accurate sound image localization. The target spatial location refers to the specific coordinates of the sound source object in the virtual acoustic environment, determined using azimuth and elevation angles in three-dimensional space. It determines the speaker positions and energy distribution that the sound source object should be assigned to in the final output multi-channel panoramic data. The azimuth angle refers to the angular position of the sound source object relative to the listener or reference point on the horizontal plane. The azimuth angle typically ranges from 0° to 360° and is used to represent the sound source object's positioning information in the left-right direction. For example, 0° represents directly in front, 90° represents directly to the right, 180° represents directly behind, and 270° represents directly to the left. The elevation angle refers to the angular position of the sound source object relative to the listener or reference point on the vertical plane. The elevation angle typically ranges from -90° to +90° and is used to represent the sound source object's positioning information in the up-down direction. For example, 0° indicates that the sound source is located in the plane directly in front, +30° indicates that the sound source is located overhead, and -30° indicates that the sound source is located on the ground. The first intensity range refers to a dynamic range used to control the gain distribution of the sound source across different speaker channels. It determines the relative loudness difference of the audio signal between different speakers, thus affecting the perceived distance and spatial immersion of the sound source.

[0189] The delay parameter is used to simulate the natural delay effect of sound propagating in space, making the sound image more realistic and three-dimensional. Understandably, the determination of the delay parameter follows the physical laws of sound wave propagation.

[0190] The delay parameter can be determined in any suitable way.

[0191] In some implementations, the delay parameter can be determined based on the ratio of the distance to the sound source to the speed of sound. The distance is the straight-line distance from the sound source (i.e., the sound source object) to the listener. The listener is a hypothetical virtual audience member, whose position can be at the center of the sound field or a specific listening point. The listener's position determines the path length of the sound to the listener and affects the delay parameter.

[0192] In some implementations, the delay parameter can be determined based on the distance to the sound source and the spatial properties of the sound source. Spatial properties can affect the time it takes for sound to travel through the environment, thus affecting the delay parameter. For example, if the sound source is located in a large concert hall, the time it takes for the sound to travel to the listener will be longer, requiring a larger delay parameter to enhance the sense of space.

[0193] Reverberation parameters may include, but are not limited to, wet-dry ratio, early reflection gain, etc.

[0194] The dry / wet ratio refers to the proportion of dry signal to wet signal in an audio signal. The dry signal is the original audio signal without reverberation processing, while the wet signal is the audio signal after reverberation processing. The setting of the dry / wet ratio directly affects the presence and integration of the sound source in space.

[0195] The wet / dry ratio can be determined in any suitable way. In some implementations, it can be determined based on diffusion. Diffusion describes the point-like or area-like characteristics of a sound source in three-dimensional space, that is, whether the audio signal of the sound source has a clear directionality (such as a human voice or lead guitar) or is more like a diffuse sound source (such as a string ensemble or ambient atmosphere). The higher the diffusion, the more blurred and integrated the audio signal is into the environment; the lower the diffusion, the more focused and clearly positioned the audio signal is. For example, in a symphony orchestra scene, the string section may be given a high diffusion to create a wide and integrated sense of space; while in a video game, the lead vocalist may be given a low diffusion to ensure that the lead vocalist's position is clear and prominent. In implementation, this diffusion is dynamically generated based on the sound source, genre, emotion, virtual acoustic environment, etc. In practice, for sound sources with high diffusion (such as choirs and background environments), the proportion of wet signal is increased to make these sound sources blend into the environment more; while for sound sources with low diffusion (such as lead singers and solo instruments), the proportion of wet signal is reduced to maintain the clear positioning and prominent expressiveness of these sound sources.

[0196] Early reflection gain refers to the intensity of early reflections in reverberation processing. Early reflections are the first reflections of an audio signal after it encounters an obstacle during propagation; they are related to the spatial and directional perception of the audio signal.

[0197] The early reflection gain can be determined in any suitable manner. In some implementations, the early reflection gain can be determined based on the diffusion. For example, for audio sources with high diffusion, the early reflection gain is increased to enhance the spatial sense and environmental blending of the audio source; while for audio tracks with low diffusion, the early reflection gain is decreased to maintain a clear sense of localization of the audio source.

[0198] Spectral parameters are used to describe the frequency attenuation of a sound source as distance changes, thereby simulating the air absorption effect of a distant sound source, making the sound source sound more natural and in line with psychological expectations, and improving the consistency and realism of the sound source at different distances.

[0199] The spectral parameters can be determined in any suitable way.

[0200] In some implementations, the spectral parameter can be determined based on the distance to the sound source object and the second intensity range corresponding to the sound source object. For example, the distance value can be normalized to between 0 and 1, where 0 represents closest to the listener and 1 represents farthest. It should be understood that distance is crucial for shaping the spatial sense of sound, as it determines the perceived distance of sound and affects subsequent processing steps such as gain control and spectrum shaping. The second intensity range characterizes the intensity intervals corresponding to different distances of the sound source object within the virtual acoustic environment in which the sound source object is located; it is a set of adjustable audio energy distribution intervals. In some implementations, the second intensity range relates to the energy distribution ratio from low to high frequencies and can be adjusted according to genre, emotion, virtual sound field environment, etc. The second intensity range is used to control the gain variation of the audio signal in different frequency bands to enhance or weaken certain frequency components, thereby achieving a more natural spatial rendering.

[0201] After determining the collaborative parameter processing, the collaborative processing parameters can be used for perception optimization processing, and then the object bitstream can be obtained in the speaker channel corresponding to the target spatial location specified by the spatial attributes.

[0202] In some implementations, where the co-processing parameters include a gain parameter, the gain of the audio source object is adjusted for different speaker channels based on this gain parameter. For example, in a 7.1.4 channel system, if the audio source object is located directly in front and in the center, the gain coefficient for the center channel corresponding to the audio source object is higher, while the gain coefficient for the left and right surround channels corresponding to the audio source object is lower. This non-uniform distribution of gain coefficients allows the user to perceive that the audio source object is coming from a specific direction.

[0203] In some implementations, where the collaborative processing parameters include spectral parameters, the frequency of the sound source object is shaped according to these spectral parameters. For example, to simulate a far-field sound source, its high frequencies are attenuated; to improve the clarity of overhead instruments, their key frequency bands are enhanced.

[0204] In some implementations, where the collaborative processing parameters include a delay parameter, delay processing is applied to the sound source object according to that delay parameter. For example, a micro-delay is added to enhance the sense of location and distance. For instance, when the sound source object is located on the left, the audio signal from the left channel arrives first, while the audio signal from the right channel arrives slightly delayed. This time difference helps guide the brain to determine the location of the sound source object, thereby enhancing the realism of spatial positioning.

[0205] In some implementations, where the co-processing parameters include reverb parameters, the sound source object is reverb-processed according to these reverb parameters. For example, on top of global reverb, a separate, lightweight, dedicated reverb is added to important objects (such as lead vocals) so that they can both blend into the environment and stand out.

[0206] It is understandable that the order in which the sound source object is processed according to the collaborative processing parameters is not limited. For example, the sound source object can be processed sequentially according to the gain parameter, spectrum parameter, delay parameter, and reverberation parameter. Alternatively, the sound source object can be processed simultaneously according to the gain parameter, spectrum parameter, delay parameter, and reverberation parameter.

[0207] In this way, customized parameters such as gain, delay, reverberation, and spectrum are generated based on the spatial attributes and processing strategies of the object's track. The sound source object is then processed accordingly, enhancing the accurate positioning of the sound source object in three-dimensional space and presenting a more realistic motion trajectory and spatial hierarchy. This makes the sound source object sound more natural and in line with psychological expectations, effectively avoiding sound field chaos. Each sound source object is integrated into the overall environment while remaining clearly distinguishable, achieving the best effect in different dimensions and significantly improving the listener's understanding of the music content and immersive experience.

[0208] Step S124: Generate a multi-channel pseudo-panoramic sound stream based on the base sound field bitstream and the object bitstream.

[0209] Here, the pseudo-immersive audio stream can be generated in any suitable way. In implementation, since the object stream may have sound image breaks in a low-density speaker layout, while the base sound field can provide continuous background filling, the two are merged to form an auditory hierarchy of "coherent background and clear foreground".

[0210] The pseudo-panoramic audio stream can be generated in any suitable way.

[0211] In some implementations, step S124 includes: mixing the base stream of each channel in the base sound field bitstream and the render stream of each channel in the object bitstream to obtain a multi-channel pseudo-immersive sound bitstream.

[0212] Here, multi-channel includes at least three of the following channels: left channel, right channel, center channel, subwoofer channel, left surround channel, right surround channel, left rear surround channel, right rear surround channel, front left top channel, front right top channel, rear left top channel, and rear right top channel.

[0213] During implementation, after allocating each rendered bitstream to its corresponding speaker channel, the rendered bitstream and the base bitstream are combined to obtain the bitstream of that speaker channel, improving the overall clarity and positioning of the audio, thereby enhancing the user's immersive listening experience.

[0214] In this way, the signals of each channel of the intelligent object stream are mixed with the corresponding channel signals of the base sound field stream to improve the overall clarity and positioning of the audio. This enables flexible compatibility and efficient enhancement of immersive audio, significantly improves the adaptability of multi-speaker playback, sound field integrity and dynamic performance, and is widely compatible with the precise control of object-oriented audio and traditional channels to form a immersive sound that is backward compatible and upward extended.

[0215] In some implementations, the base sound field bitstream and the object bitstream can be input into a third generation model to obtain the pseudo-panoramic sound bitstream. This third generation model can be any suitable neural network model capable of performing this function.

[0216] In this embodiment, firstly, by making fusion decisions on reverberation, category, and various sound source objects, a unique sound field psychological model is dynamically generated for the audio, achieving multi-dimensional perception fusion, constructing an emotionally resonant sound field, optimizing cognitive comfort, and enabling deep personalization and contextual adaptation of the listening experience, significantly improving the emotional resonance, scene immersion, and cognitive comfort of the audio content. Secondly, a base sound field bitstream is generated using global sound field attributes and raw audio data to simulate the target virtual acoustic environment, providing a sense of spatial enclosure for the entire sound field. Thirdly, object bitstreams are generated using the spatial attributes and processing strategies of each object track, enabling efficient compression and flexible reconstruction of sound field information, significantly improving bandwidth utilization, spatial immersion, and terminal adaptability of audio transmission. Finally, by mixing the base sound field bitstream and object bitstreams, collaborative processing of environmental perception and object rendering is achieved, thereby maintaining clear sound image localization while enhancing spatial enclosure.

[0217] In some embodiments, step S13 includes steps S131 and S132, wherein: Step S131: Determine the gain coefficient of each sound source object based on the acoustic parameters corresponding to the current usage scenario.

[0218] Here, acoustic parameters may include, but are not limited to, seat position, interior space dimensions (such as cabin dimensions), sound absorption characteristics of interior materials, and acoustic performance characteristics of each speaker at each listener.

[0219] The gain coefficient can be determined in any suitable way.

[0220] In some implementations, step S131 includes: determining the ideal signal for each speaker based on the acoustic parameters corresponding to the current usage scenario and the hardware parameters of each speaker; and for each speaker, reconstructing the bitstream of the corresponding channel of the speaker based on the ideal signal of the speaker to obtain the gain coefficient of the audio source object corresponding to the speaker.

[0221] Here, hardware parameters may include, but are not limited to, frequency response range, sensitivity, impedance, amplifier output sensitivity, and rated power. Frequency response range refers to the range of frequencies a speaker or loudspeaker can reproduce. Sensitivity refers to the sound pressure level produced by a speaker at a distance of 1 meter with an input of 1 watt of power. Impedance refers to the speaker's resistance to current; impedance is not equal to sound quality, but it must be matched to the amplifier output. Amplifier output sensitivity refers to the input voltage required for the amplifier to reach its maximum rated output power. Rated power refers to the average power that the device can continuously output under undistorted, long-term safe operating conditions.

[0222] The ideal signal can be a reference signal, and it can include, but is not limited to, volume, delay, phase, and frequency response. The ideal signal can be determined in any suitable way.

[0223] In some implementations, a correspondence between each acoustic parameter, each hardware parameter, and each ideal signal can be established in advance. Based on this correspondence, an ideal signal that is compatible with both the acoustic and hardware parameters can be obtained.

[0224] In some implementations, acoustic parameters and various hardware parameters can be input into a pre-established fourth generation model to obtain ideal signals. This fourth generation model can be any suitable neural network model capable of achieving this function.

[0225] Reconstruction processing uses an ideal signal as a reference and is supplemented by subjective listening results to tune the bitstream of the audio channels, thereby achieving a more accurate, higher-quality, and personalized sound field reconstruction. Tuning processing can include, but is not limited to, frequency domain processing and time domain processing.

[0226] In this way, on the one hand, the reference signal for each speaker is determined based on the acoustic parameters of the usage scenario and the hardware parameters of each speaker. This dual-parameter driven reference signal setting can accurately match the speaker capabilities and compensate for the acoustic defects of the usage scenario, making the sound both accurate and stable. It achieves the physical accuracy of the sound field reproduction and the optimal matching of system performance, significantly improving clarity, dynamic range, and long-term stability. On the other hand, the panoramic sound bitstream is reconstructed based on the ideal signal corresponding to the speaker to generate the gain of the corresponding sound source object. This achieves the purpose of reverse mapping the physical acoustic target to the object-level rendering, realizing the WYSIWYG sound field construction, accurately reproducing the sound field and providing personalized listening experience, improving spatial consistency, dynamic response capability, and multi-scene adaptability.

[0227] In some implementations, a correspondence between each acoustic parameter, each sound source object, and each gain coefficient can be established in advance. Based on this correspondence, the gain coefficient of each sound source object that is adapted to the acoustic parameter can be obtained.

[0228] Step S132: For each channel in the multi-channel system, determine the target loudspeaker corresponding to the channel, generate the target bitstream of the channel based on the gain coefficient of the audio source object corresponding to the channel and the bitstream of the channel, and use the target bitstream of the channel as the driving signal of the target loudspeaker corresponding to the channel.

[0229] Here, different channels can correspond to different speakers. The target speaker can be determined in any suitable way. In some implementations, a correspondence between each channel and each speaker can be established in advance, and the target speaker corresponding to that channel can be obtained based on this correspondence. For example, the left channel can correspond to the front left speaker (such as the front left super tweeter, front left tweeter, front left midrange speaker, front left woofer, etc.) and the rear left speaker (such as the rear left super tweeter, rear left tweeter, rear left midrange speaker, rear left woofer, rear left headrest left and right midrange speakers, etc.); the right channel can correspond to the front right speaker (such as the front right super tweeter, front right tweeter, front right midrange speaker, front right woofer, etc.) and the rear right speaker (such as the rear right super tweeter, rear right tweeter, rear right midrange speaker, rear right woofer, rear right headrest left and right midrange speakers, etc.).

[0230] In implementation, since there can be at least one audio source object corresponding to a channel, that is, different audio source objects can correspond to the same channel, the target bitstream of the channel includes at least one channel. The number of target bitstreams is the same as the number of audio source objects. Different target bitstreams of the same channel can correspond to different speakers.

[0231] The target bitstream for this channel can be determined in any suitable way.

[0232] In some implementations, the target bitstream for a channel can be generated based on the product of the gain coefficient of the corresponding audio source object and the bitstream of the channel. For example, the product, a weighted average of the product, etc., can be used as the target bitstream. It is understood that this product represents the equivalent data strength of the signal after gain adjustment, reflecting the effective audio information carrying capacity after amplification or attenuation per unit time. Although the bitstream itself remains unchanged, the gain coefficient alters the actual audible intensity of the signal. Thus, by using gain compensation to partially compensate for sound quality loss, the sound becomes clearer and more discernible, improving the consistency and stability of listening experience in weak signal environments, achieving an efficient balance between bitstream and listening experience, and effectively improving the user experience.

[0233] In some implementations, the gain coefficient of the audio source object corresponding to the channel and the bitstream of the channel can be input into a pre-established bitstream generation model to obtain the target bitstream. This bitstream generation model can be any suitable neural network model capable of achieving this function.

[0234] In some implementations, for speaker characteristics, such as the 65-100Hz frequency components of the mid-range channel that should be represented by the center speaker, due to the bandwidth limitations of the vehicle's center speaker, these components are allocated to the left and right woofers for low-frequency compensation.

[0235] The following example illustrates the process using a 12-channel pseudo-Atmos stream in 7.1.4 format and a 53-speaker system: If two audio source objects correspond to the left channel, then the target bitstream of the left channel includes the first target bitstream of the left channel and the second target bitstream of the left channel. The first target bitstream of the left channel can be assigned to the front left speaker, and the second target bitstream of the left channel can be assigned to the rear left speaker. If two audio source objects correspond to the right channel, then the target bitstream of the right channel includes the first target bitstream of the right channel and the second target bitstream of the right channel. The first target bitstream of the right channel can be assigned to the front right speaker, and the second target bitstream of the right channel can be assigned to the rear right speaker. If the four audio source objects correspond to the center channel, then the target bitstream of the center channel includes the first target bitstream of the center channel, the second target bitstream of the center channel, the third target bitstream of the center channel, and the fourth target bitstream of the center channel. The first target bitstream of the center channel can be allocated to the front center two-way crossover (1 tweeter + 1 midrange speaker), the second target bitstream of the center channel can be allocated to the middle center two-way crossover (1 tweeter + 1 midrange speaker), the third target bitstream of the center channel can be allocated to the front left woofer and the front right woofer, and the fourth target bitstream of the center channel can be allocated to the rear left woofer and the rear right woofer. If two audio source objects correspond to the subwoofer channel, then the target bitstream of the subwoofer channel includes a first target bitstream of the subwoofer channel and a second target bitstream of the subwoofer channel. The first target bitstream of the subwoofer channel can be assigned to the subwoofer speaker, and the second target bitstream of the subwoofer channel can be assigned to the driver's seat exciter*2, the passenger seat exciter*2, the left rear seat exciter*2, and the right rear seat exciter*2. If two audio source objects correspond to the left surround channel, then the target bitstream of the left surround channel includes the first target bitstream of the left surround channel and the second target bitstream of the left surround channel. The first target bitstream of the left surround channel can be assigned to the rear left speaker, and the second target bitstream of the left surround channel can be assigned to the left surround tweeter and the left surround midrange speaker. If two audio source objects correspond to the right surround channel, then the target bitstream of the right surround channel includes the first target bitstream of the right surround channel and the second target bitstream of the right surround channel. The first target bitstream of the right surround channel can be assigned to the rear right speaker, and the second target bitstream of the right surround channel can be assigned to the right surround tweeter and the right surround midrange speaker. If the three audio source objects correspond to the left rear surround channel, then the target bitstream of the left rear surround channel includes the first target bitstream of the left rear surround channel, the second target bitstream of the left rear surround channel, and the third target bitstream of the left rear surround channel. The first target bitstream of the left rear surround channel can be assigned to the left surround tweeter and the left surround midrange speaker. The second target bitstream of the left rear surround channel can be assigned to the rear left woofer. The third target bitstream of the left rear surround channel can be assigned to the left and right midrange speakers of the driver's headrest *2. If the three audio source objects correspond to the right rear surround channel, then the target bitstream of the right rear surround channel includes the first target bitstream of the right rear surround channel, the second target bitstream of the right rear surround channel, and the third target bitstream of the right rear surround channel. The first target bitstream of the right rear surround channel can be assigned to the right surround tweeter and the right surround midrange speaker. The second target bitstream of the right rear surround channel can be assigned to the rear right woofer. The third target bitstream of the right rear surround channel can be assigned to the left and right midrange speakers of the passenger headrest*2. If two audio source objects correspond to the front left top channel, then the target bitstream of the front left top channel includes the first target bitstream of the front left top channel and the second target bitstream of the front left top channel. The first target bitstream of the front left top channel can be assigned to the front left top tweeter and midrange speaker, and the second target bitstream of the front left top channel can be assigned to the front left woofer. If two audio source objects correspond to the front right top channel, then the target bitstream of the front right top channel includes the first target bitstream of the front right top channel and the second target bitstream of the front right top channel. The first target bitstream of the front right top channel can be assigned to the front left top tweeter and midrange speaker, and the second target bitstream of the front right top channel can be assigned to the front right woofer. If two audio source objects correspond to the rear left top channel, then the target bitstream of the rear left top channel includes the first target bitstream of the rear left top channel and the second target bitstream of the rear left top channel. The first target bitstream of the rear left top channel can be assigned to the rear left top tweeter and midrange speaker, and the second target bitstream of the rear left top channel can be assigned to the rear left woofer. If two audio source objects correspond to the rear right top channel, then the target bitstream of the rear right top channel includes the first target bitstream of the rear right top channel and the second target bitstream of the rear right top channel. The first target bitstream of the rear right top channel can be assigned to the rear right top tweeter and midrange speaker, and the second target bitstream of the rear right top channel can be assigned to the rear right woofer.

[0236] In this embodiment, firstly, the gain coefficient of each object track is determined based on the acoustic parameters of the usage scenario, significantly improving the clarity, spatial balance, and personalized listening experience of the audio content in a specific environment. Compared to a fixed gain strategy, dynamic adjustment can precisely compensate for acoustic defects in the usage scenario, allowing each sound object to be properly positioned. Secondly, the speakers corresponding to each channel are dynamically configured. Compared to a fixed channel layout, dynamic configuration breaks the limitations of physical placement, allowing the sound to follow the environment and achieving a more intelligent and immersive listening experience. Finally, the bitstream of the corresponding channel is compensated based on the gain coefficient of each sound source object, achieving the goal of combining the flexibility of objectified audio with the stability of the channel system. This ensures that each sound element can "be in its proper place and make its own sound" in three-dimensional space, realizing dynamic sound field balance and precise optimization of listening experience, significantly improving the clarity, spatial layering, and personalized adaptability of audio in complex environments.

[0237] Based on the above embodiments, this application also provides an audio processing system, such as... Figure 2 As shown, the audio processing system 20 includes an audio input module 21, an extraction module 22, a mixing module 23, a reconstruction module 24, and an audio output module 25, wherein: Audio input module 21 is used to receive raw audio data; Extraction module 22, connected to audio input module 21, is used to determine key information of the original audio data. Key information includes multiple sound source objects, category information and reverberation information. Category information represents the genre to which the original audio data belongs and / or the emotion of the original audio data. The mixing module 23 is connected to the audio input module 21 and the extraction module 22 respectively, and is used to generate a multi-channel pseudo-immersive audio stream based on multiple sound source objects, category information and reverberation information; The reconstruction module 24, connected to the mixing module 23, is used to determine the driving signals of multiple target speakers in the current usage scenario based on the acoustic parameters corresponding to the current usage scenario and the multi-channel pseudo-immersive audio bitstream. The audio output module 25, connected to the reconstruction module 24, is used to drive each target speaker to play based on the drive signal of each target speaker.

[0238] Here, the audio input module can be any suitable module capable of receiving audio data. This audio input module can receive audio files of any format, such as mp3, FLAC, WAV, etc.

[0239] The extraction module can be any suitable module capable of performing this function. The process by which the extraction module determines key information can be found in the specific implementation of step S11 above.

[0240] The mixing module can be any suitable module capable of performing this function. The process by which the mixing module generates the pseudo-Atmos bitstream can be found in the specific implementation of step S12 above.

[0241] The reconfiguration module can be any suitable module capable of performing this function. The process by which the reconfiguration module determines each driving signal can be found in the specific implementation of step S13 above.

[0242] The audio output module can be any suitable module capable of signal output. In implementation, the audio output module converts the analog drive signal into a corresponding digital signal for output.

[0243] In this embodiment, firstly, the extraction module separates different sound source objects, genres, emotions, reverb, etc., from the original audio data, achieving a deep understanding of the audio content and greatly improving the separation quality. This lays the foundation for developing differentiated rendering strategies for audio of different styles, enriching the processing effects and reducing the possibility of unnatural sound fields due to low separation quality. Secondly, the mixing module generates a panoramic sound bitstream based on the sound source objects, genres, emotions, reverb information, etc. In other words, it can upmix two-channel stereo audio to output high-quality multi-dimensional panoramic sound audio with immersion, clear positioning, and content adaptability. This enriches the mixing processing methods, allows for customized adjustments based on the specific characteristics of the audio content, improves the mixed effect, and reduces the possibility of blurred sound fields, false spatial sense, and chaotic instrument positioning caused by rigid or monotonous mixing processing methods, achieving a truly immersive high-quality listening experience. Finally, the reconstruction module determines and outputs the drive signals for each speaker based on the current usage scenario and the panoramic sound bitstream. In other words, it can perform precise sound field reconstruction based on the characteristics of the audio and the usage environment, providing listeners with higher quality and more personalized sound effects, thereby greatly improving the listening experience.

[0244] In some implementations, the extraction module 22 is used to separate the original audio data using a pre-trained separation model to obtain multiple sound source objects; and to analyze and process the original audio data to obtain category information and reverberation information.

[0245] In some implementations, the separation model includes an encoding module, a separation module, and a decoding module; the extraction module 22 is used to preprocess the original audio data to obtain the complex spectrum corresponding to the original audio data; the encoding module is used to encode the complex spectrum to obtain a first feature; the separation module is used to perform time-frequency domain modeling on the first feature to obtain a second feature; and the decoding module is used to decode the second feature to obtain multiple sound source objects.

[0246] In some implementations, the complex spectrum is used to characterize the frequency range of the speech band in the original audio data; the extraction module 22 is used to divide the frequency range of the speech band using a preset sparse banding strategy to obtain multiple first sub-bands; and each first sub-band is subjected to downsampling compression processing at a corresponding multiple to obtain a first feature.

[0247] In some implementations, the separation module includes a frequency domain processing unit, a weighted processing unit, and a time domain processing unit; the extraction module 22 is used to use the frequency domain processing unit to perform frequency band modeling on the first feature to obtain a third feature; use the weighted processing unit to perform weighted processing on the first feature and the third feature to obtain a fourth feature; use the time domain processing unit to perform frame number modeling on the fourth feature to obtain a fifth feature; and use the weighted processing unit to perform weighted processing on the fourth feature and the fifth feature to obtain a second feature.

[0248] In some implementations, the extraction module 22 is used to divide the frequency range of the second feature using a preset sparse banding strategy to obtain multiple second sub-bands; and to perform upsampling expansion processing on each second sub-band at a corresponding multiple to obtain multiple sound source objects.

[0249] In some implementations, the system further includes a training module, which is used to input training samples into the separation model to be trained to obtain multiple predicted sound source objects; determine a target loss value based on the label information corresponding to the multiple predicted sound source objects and the training samples; and update the model parameters of the separation model at least once based on the target loss value to obtain the trained separation model.

[0250] In some implementations, the extraction module 22 is used to: input the raw audio data into a pre-trained first analysis model to obtain category information and reverberation information; or input the raw audio data and multiple sound source objects into a pre-trained second analysis model to obtain category information and reverberation information.

[0251] In some implementations, the mixing module 23 is used to determine a sound field effect model based on reverberation information, category information, and multiple sound source objects; wherein the sound field effect model includes global sound field attributes, spatial attributes of each sound source object, and processing strategies for each sound source object; a base sound field bitstream is generated based on the global sound field attributes and the original audio data; an object bitstream is generated based on the spatial attributes and processing strategies of each sound source object; and a multi-channel pseudo-immersive sound bitstream is generated based on the base sound field bitstream and the object bitstream.

[0252] In some implementations, the mixing module 23 is used to determine the virtual acoustic environment corresponding to the original audio data based on reverberation information; and to determine the sound field effect model from a preset rule base based on category information, multiple sound source objects and the virtual acoustic environment.

[0253] In some implementations, the mixing module 23 is used to determine a reverberation processing strategy and an equalization processing strategy based on global sound field properties; to perform reverberation processing on the original audio data according to the reverberation processing strategy to obtain first audio data; and to perform equalization processing on the first audio data according to the equalization processing strategy to obtain a base sound field bitstream.

[0254] In some implementations, the object bitstream includes the rendered bitstream of the speaker channel corresponding to each sound source object; the mixing module 23 is used to determine the co-processing parameters of the sound source object based on the spatial attributes and processing strategy of the sound source object for each sound source object, and generate the rendered bitstream of the speaker channel corresponding to the sound source object based on the co-processing parameters; wherein, the co-processing parameters include at least one of the following: gain parameter, delay parameter, reverberation parameter, and spectrum parameter.

[0255] In some implementations, the mixing module 23 is used to mix the base stream of each channel in the base sound field bitstream and the render stream of each channel in the object bitstream to obtain a multi-channel pseudo-Atmos bitstream; wherein, the multi-channel includes at least three channels selected from the following: left channel, right channel, center channel, subwoofer channel, left surround channel, right surround channel, left rear surround channel, right rear surround channel, front left top channel, front right top channel, rear left top channel, and rear right top channel.

[0256] In some implementations, the reconstruction module 24 is used to determine the gain coefficient of each sound source object based on the acoustic parameters corresponding to the current usage scenario; for each channel in the multi-channel, determine the target loudspeaker corresponding to the channel; generate the target bitstream of the channel based on the gain coefficient of the sound source object corresponding to the channel and the bitstream of the channel; and use the target bitstream of the channel as the driving signal of the target loudspeaker corresponding to the channel.

[0257] In some implementations, the reconstruction module 24 is used to determine the ideal signal corresponding to each speaker based on the acoustic parameters corresponding to the current usage scenario and the hardware parameters of each speaker; for each speaker, based on the ideal signal corresponding to the speaker, the bitstream of the corresponding channel of the speaker is reconstructed to obtain the gain coefficient of the audio source object corresponding to the speaker.

[0258] In some implementations, the reconstruction module 24 is used to generate a target bitstream for a channel based on the product between the gain coefficient of the audio source object corresponding to the channel and the bitstream of the channel.

[0259] The description of the system embodiments above is similar to that of the method embodiments above, and has similar beneficial effects. For technical details not disclosed in the system embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0260] The technical solution of this application will be described in detail below using a vehicle as an example of its current application scenario. It is understood that the technical solution of this application can also be applied to home theaters, cinemas, and other similar application scenarios.

[0261] With the rapid development of artificial intelligence (AI) technology, its application in the music field is becoming increasingly widespread and in-depth, bringing new breakthroughs and possibilities to music source separation and music genre classification. The advancements and achievements in this field have also brought new opportunities to in-vehicle audio technology, which faces increasingly strong user entertainment demands and more audio transmission channels. Utilizing information such as music sources and music tags separated by AI technology, combined with the entire vehicle speaker system, to design sound effect algorithms can bring added value to in-vehicle audio experiences. Music source separation has always been one of the important challenges in the field of audio processing. Traditional methods are often based on specific assumptions and prior knowledge, and their effectiveness is limited when facing complex music mixtures. However, with the rise of deep learning algorithms, especially the application of models such as Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN), it has become possible to separate individual sound sources from complex mixed music. The technical principles typically involve the following aspects: First, the audio signal is converted into a time-frequency representation, such as a spectrogram, through methods such as Short-Time Fourier Transform (STFT). Then, deep learning models, such as convolutional neural networks, are used to extract and learn features from these time-frequency representations. Convolutional layers capture local frequency and temporal patterns, while pooling layers reduce feature dimensionality and extract key features. Recurrent neural networks can process time-series information in audio signals and capture long-term dependencies. Trained on a large amount of labeled music data, the model learns the feature differences of different sound sources in the time-frequency domain, thus enabling the separation of new mixed music.

[0262] Meanwhile, the classification of music genres is also crucial for music management, recommendation, and understanding. Traditional classification methods typically rely on manual annotation and rule-based judgment, which are not only time-consuming and labor-intensive but also highly subjective. The introduction of AI (Artificial Intelligence) provides a more objective and automated solution for music genre classification. The technical principles of AI music genre classification mainly include feature extraction and the application of classification models. For feature extraction, various features are extracted from the audio signal of the music. For example, melody features can be obtained through pitch sequence analysis; rhythm features can be obtained through calculation of beats, note durations, etc.; harmony features involve chord analysis; and timbre features are related to the distribution of the spectrum. The extracted features are input into the classification model. Common classification models include Support Vector Machines (SVM), decision trees, random forests, and convolutional neural networks in deep learning. These models, by learning the feature patterns of known music genres, can predict and classify new musical works according to their genre.

[0263] Traditionally, music transmission relies on stereo audio technology. However, the development of multi-channel audio applications has spurred the expansion of in-vehicle speakers and speaker channels in the automotive audio field. Existing stereo upmixing technologies often employ fixed acoustic models or simple panning rules, resulting in rigid processing methods that cannot adaptively adjust to the specific characteristics of the audio content. This leads to common problems in the mixed audio, such as a blurred sound field, artificial spatial sense, and chaotic instrument positioning, making it difficult to achieve a truly immersive, high-quality listening experience. While some advanced solutions have introduced source separation technology, these typically only simply assign the separated elements to different channels, lacking a deep understanding of the audio content and failing to develop differentiated rendering strategies for different music styles. The processing effect is monotonous, and the sound field can easily become unnatural due to poor separation quality. Traditional in-vehicle audio systems usually use fixed equalizer settings and simple surround sound processing, unable to perform personalized and precise sound field reconstruction based on the characteristics of the music and the in-vehicle environment, resulting in a less than ideal sound experience. Multichannel audio sources are primarily used in film audio distribution. Filmmakers, using tools like 7.1.4 channel design tools, design the effects of different sound objects on playback systems to create mixed audio, which is then decoded by playback devices and played back on multichannel systems using multichannel audio transmission technology. Another area of ​​multichannel audio is music playback, which also has two branches: The first is similar to film scoring, providing design tools to recording and mixing engineers and encouraging musicians and labels to produce music in multichannel transmission formats. This method has limitations, requiring mixing engineers to create innovative mixes specifically for multichannel audio transmission technology. Current music libraries are limited, and it cannot create mixing effects for in-car audio playback systems. The second method involves car manufacturers using virtual surround sound mixing on traditional stereo sources. This method utilizes virtual sound imaging technology that plays the same music source through multiple speakers, but the experience is limited.

[0264] Therefore, there is an urgent need for a solution that can intelligently analyze audio content and adaptively generate in-vehicle panoramic sound audio with accurate sound image positioning and rich spatial layering. This solution combines the two methods mentioned above. By combining the source audio and genre characteristics of AI music source separation and extraction and AI music genre recognition with the playback characteristics of the real vehicle multi-channel audio system, a sound effect algorithm is designed to remix the audio in the role of a "mixing engineer" and reconstruct the in-vehicle sound field, providing users with a more immersive listening experience.

[0265] This application proposes a four-dimensional in-vehicle audio effect method and system for in-vehicle sound field reconstruction using AI music source separation and AI music genre classification. The purpose of this application is to provide an in-vehicle audio effect method and system that utilizes AI music source separation and AI music genre classification to reconstruct the in-vehicle sound field, addressing the problems of existing in-vehicle audio systems. It upmixes traditional two-channel stereo audio into four-dimensional panoramic sound (i.e., immersive audio technology combining three-dimensional spatial and temporal dimensions) and reconstructs the sound field within the vehicle, providing an intelligent and adaptive four-dimensional panoramic sound mixing solution. Its core objective is not only to achieve audio track separation and allocation, but also to deeply understand the music content (such as genre, emotion, and acoustic environment) through multimodal AI technology, and dynamically generate an optimal "sound field effect psychological model" based on this. Finally, through a model-driven collaborative processing chain, it outputs high-quality panoramic sound audio that combines immersion, clear positioning, and content adaptability, providing in-vehicle passengers with a higher quality and more personalized audio experience.

[0266] In some implementations, the extraction module may include an AI audio separation module and an AI music classification module; the mixing module may include a dual-channel stereo audio intelligent mixing to a multi-channel immersive sound module, an in-vehicle pseudo-immersive sound audio mixing and playback processing module, and an in-vehicle multi-channel audio system and upmixing module; the reconstruction module may include an in-vehicle acoustic analysis module, a sound field reconstruction calculation module, and a tuning system module with multiple functions and interfaces. For example... Figure 3 As shown, the four-dimensional sound in-vehicle audio system (corresponding to the aforementioned audio processing system) that uses AI music source separation and AI music genre classification to reconstruct the in-vehicle sound field includes: an audio input module, an AI audio separation module, an AI music classification module, a dual-channel stereo audio intelligent mixing to multi-channel panoramic sound module, an in-vehicle pseudo-panoramic sound audio mixing and playback processing module, an in-vehicle multi-channel audio system and upmixing module, an in-vehicle acoustic analysis module, a sound field reconstruction calculation module, an audio output module, a tuning system module with multiple functions and interfaces, and a multi-channel in-vehicle audio system.

[0267] The audio input module is simultaneously connected to the AI ​​audio separation module and the AI ​​music classification module. The in-vehicle multi-channel audio system and upmixing module are connected to the in-vehicle acoustic analysis module. The in-vehicle acoustic analysis module is connected to the sound field reconstruction calculation module. The sound field reconstruction calculation module is connected to the tuning system module with multiple functions and interfaces. The tuning system module with multiple functions and interfaces is connected to the audio output module. In some embodiments, this four-dimensional sound in-vehicle audio system may also include a user interaction module, which is connected to the audio input module, the in-vehicle multi-channel audio system, and the upmixing module.

[0268] The audio input module supports multiple audio input methods, including MP3, FLAC, and WAV audio sources, such as online streaming media, Bluetooth connection, USB interface, and AUX input, to acquire music played by the user. The audio input module receives the raw stereo signal (corresponding to the aforementioned raw audio data) and performs preprocessing such as standardization and noise reduction.

[0269] The AI ​​audio separation module and the AI ​​music classification module perform deep analysis on the input audio and extract multi-dimensional features. They are two modules that work in parallel. (1) The AI ​​audio source separation module uses a pre-trained deep learning model (corresponding to the aforementioned separation model) to separate the input audio into multiple independent audio source object tracks with high precision, including but not limited to vocals, lead instruments, bass, drum kits, backing vocals, etc. Each object track has its type label.

[0270] (2) AI music classification module: analyzes the input audio in parallel, analyzes its musical features and classifies it.

[0271] The dual-channel stereo audio intelligent mixing to multi-channel immersive sound module includes: (1) Dynamic sound field psychological model generation module, which receives all information from the above analysis module (i.e., AI audio separation module + AI music classification module) and performs optimal sound field construction and mixing strategy for different music attribute combinations in standard immersive sound environment.

[0272] (2) Adaptive multi-track rendering and synthesis engine, responsible for the final sound rendering, generating intelligent object rendering audio stream and base sound field generation audio stream, and finally mixing the channel signals of each channel of intelligent object rendering stream (i.e., intelligent object rendering audio stream) with the corresponding channel signals of base sound field generation stream (i.e., base sound field generation audio stream) to output the final multi-channel pseudo-immersive audio stream under standard environment (corresponding to the aforementioned pseudo-immersive audio stream).

[0273] The in-vehicle pseudo-immersive sound audio mixing and playback processing module compares and calculates the differences between the in-vehicle speaker system and the standard immersive sound playback environment, processes the pseudo-immersive sound audio streams with a certain gain ratio, and sends them to each speaker channel.

[0274] The in-vehicle acoustic acquisition and analysis module acquires acoustic parameters of the in-vehicle space through sensors arranged in the vehicle, including seat position, cabin size, sound absorption characteristics of interior materials, and acoustic performance characteristics of each speaker in each passenger seat.

[0275] The sound field reconstruction calculation module combines the actual vehicle speaker's frequency response, sensitivity, impedance, amplifier output sensitivity, rated power, and other hardware characteristics, as well as the collected in-vehicle acoustic parameters, to calculate the ideal output signal for each speaker, including volume, delay, phase, frequency response, etc.

[0276] The audio is processed through a digital signal processing module to achieve a more accurate, higher-quality, and personalized sound field reconstruction.

[0277] The audio output module drives the multi-channel speaker system in the vehicle, including the front doors, rear doors, center console, roof, seats, headrests, and other locations, based on the calculated output signal, to achieve all-around sound coverage and create a personalized and high-quality sound effect for passengers in the vehicle.

[0278] The digital signal processing module is the hardware that runs all the algorithms of the above digital signal processing modules, including but not limited to: CPU, GPU, DSP, and MCU.

[0279] The user interaction module uses the in-vehicle display screen to show information about the currently playing music, such as song title, artist, and genre, as well as provide sound effect settings switches and options for users to further process and compare the effects of separate music elements.

[0280] The following is a detailed introduction to each module: The audio input module supports multiple audio input methods and audio format sources such as mp3, FLAC, and WAV, including online streaming media, Bluetooth connection, USB interface, and AUX input, to obtain the music played by the user. After the user clicks play through the interactive interface, the audio data is decoded by the player and preprocessed through standardization, noise reduction, etc., to obtain the raw audio bitstream data to be processed.

[0281] The AI ​​audio separation module utilizes a pre-trained deep learning model for processing. This application proposes a method for separating, but not limited to, vocals, lead instruments, bass, drum kits, and backing vocals. Each object track is labeled with its type. The deep convolutional neural network design for the music source is as follows: like Figure 4 As shown, the neural network architecture consists of three parts: an encoder module 31, a separation module 32, and a decoder module 33. The encoder module is responsible for extracting the feature information of the original audio, the separation module is used for time-frequency domain modeling of the signal, and the decoder module reconstructs the modeled features into a time-domain signal. The specific implementation is as follows: (1) Encoder module The model's initial input is a time-domain mixed signal. After Short-Time Fourier Transform (STFT), the complex spectrum of the mixed signal is fed into the encoder module. The encoder module consists of multiple stacked encoder layers (convolutional layers, normalization layers, and activation functions). A sparse banding strategy is employed, decomposing the speech audio bands into three sub-bands (low frequency, mid frequency, and high frequency) and downsampling and compressing them at different ratios (1x, 4x, and 16x). Through continuous convolutional downsampling, the encoder module reduces the model's frequency dimension and computational complexity while extracting high-dimensional features suitable for the separation task.

[0282] (2) Separation module The separation module mainly performs time-frequency domain modeling on the high-dimensional features extracted by the encoder module. It consists of a gated attention mechanism (GAU) and a bidirectional long short-term memory network (BLSTM) stacked together, and each GAU and BLSTM contains both a frequency domain path and a time domain path (dual-path structure).

[0283] Dual-path GAU and BLSTM achieve differentiated attention to the importance of musical features by assigning dynamic weights to different time and frequency positions: for spectrally rich segments (such as the chorus) or key frequency ranges, the model automatically assigns higher weights to enhance the priority of these regions in the separation process, thereby improving the effectiveness of music source separation. Simultaneously, a multi-head attention mechanism captures multi-dimensional dependencies and importance in the input sequence, further enhancing the model's ability to represent and process complex musical data.

[0284] (3) Decoder module The decoder module can be viewed as the inverse process of the encoder module. It restores the high-dimensional features modeled by the separation module to the complex spectrum form of the separated signal through convolutional upsampling, and then obtains the time-domain sampled signals of each instrument track through inverse short-time Fourier transform (ISTFT). The decoder module consists of multiple stacked decoder layers (transposed convolutional layers, normalization layers, and activation functions). A sparse banding strategy is employed, decomposing the speech audio band into three sub-bands (low frequency, mid frequency, and high frequency) and upsampling them at different ratios (1x, 4x, and 16x).

[0285] For newly built deep convolutional neural networks, it is necessary to use a database and set the corresponding loss function and other parameters to optimize learning efficiency, avoid overfitting, and improve separation accuracy. The process consists of steps a through e, where: a. Loss Function Setting The loss function employs a composite loss function that integrates time-frequency domain information, considers both amplitude and phase characteristics, and incorporates an adaptive weighting mechanism. This allows for a more comprehensive measurement of the difference between the estimated signal and the target signal, thereby improving model performance. The loss function is achieved through a weighted fusion of three sub-losses: a frequency domain complex spectrum loss, and a loss function that integrates time-frequency domain information, considering both amplitude and phase characteristics. SDR normalized loss and temporal compression loss ; For frequency domain complex spectrum loss The solution includes the following steps 1 to 3, wherein: Step 1: Complex spectrum and amplitude calculation; Pure complex spectrum: Estimate the complex spectrum: ;in, It is a model that estimates speech (time domain). It is the pure target speech (time domain, i.e., label information); Short-Time Fourier Transform (STFT) outputs a complex spectrum. , It is the number of effective frequency points.

[0286] Pure amplitude spectrum: Estimated amplitude spectrum: ;in, These are the real and imaginary parts of the complex spectrum, respectively. It is a numerically stable small constant.

[0287] Step 2: Spectrum compression (polar coordinate normalization + compression); , , , ;in, It is the spectral compression coefficient.

[0288] Step 3: Complex spectrum loss function; ;in, It is the global mean; It is the L2 norm; It is the amplitude / phase loss weight.

[0289] SDR Normalized Loss as follows: ;in, It is the global mean; It is an L2 norm; It is the amplitude / phase loss weight; It is a model that estimates speech; It is a pure target voice; It is a pure complex spectrum; It is to estimate the complex spectrum; It is a pure amplitude spectrum; It is an estimation of the amplitude spectrum.

[0290] Temporal compression loss The solution includes the following steps 1 and 2, wherein: Step 1: Time-domain amplitude compression; Pure time-domain amplitude compression: Estimated time-domain amplitude compression: ;in, ;in, It is the spectral compression coefficient; It is a numerically stable small constant; It is a model that estimates speech; It is the pure target voice.

[0291] Step 2: Temporal Loss Function ; ;in, It is the global mean.

[0292] So, the total loss function (Corresponding to the aforementioned objective loss function) is:

[0293] in, , , Adjustments are made based on actual test set performance to balance the contributions of different sub-losses. In some implementations, .

[0294] In some implementations, normalization can also be performed. If normalization is enabled, the signal peak value is first normalized. ,in, ; It is a numerically stable small constant; It is a model that estimates speech; It is the pure target voice.

[0295] b. Optimizer settings: Select the Adam optimizer, with an initial learning rate of 0.0005, and adopt a learning rate decay strategy as training progresses.

[0296] c. Data Selection This application uses the open-source music track separation training set Musdb18, which contains more than 150 full-length music tracks of different genres (lasting approximately 10 hours), along with separate recordings of vocals, piano, drums, bass, etc. To increase the variety of separated music sources, 150 mixing tracks and subjectively selected open-source music source separation tools were added as supplements to the training set.

[0297] d. Training epochs setting: Set to 200 epochs based on the dataset size and model convergence.

[0298] e. Batch Size Setting: For training based on two H20 graphics cards, the Batch Size is set to 32.

[0299] The AI ​​music classification module uses a pre-trained deep learning model (corresponding to the aforementioned first analysis model) to analyze the input audio. This sub-module contains the following units: Genre and Emotion Recognition Unit: Based on the pitch sequence, rhythmic features, harmonic progressions, and spectral energy distribution of music, it identifies music genres (such as classical, rock, jazz, and electronic) through a classification neural network and infers their emotional characteristics (such as exciting, soothing, and melancholic).

[0300] This application proposes a deep convolutional neural network design for music genres such as blues, classical, country, disco, hip hop, jazz, metal, pop, reggae, and rock: The model takes the Mel spectrum as input. Different music genres have some special differences in their Mel spectrograms. The model cleverly transforms the music genre classification problem into an image classification task.

[0301] The specific implementation is as follows: This CNN network consists of stacked 2D convolutional layers, max pooling layers, fully connected layers, dropout layers, and activation functions. The 2D convolutional layers are used for feature extraction, the max pooling layers are used for downsampling, the fully connected layers are used to capture global information, dropout is used to prevent the model from overfitting, and the final output layer uses the softmax activation function for classification problems.

[0302] The training configuration for the music genre classification model is as follows: Dataset: GTZAN open source dataset, which includes 10 different genres (blues, classical, country, disco, hip hop, jazz, metal, pop, reggae and rock), with 100 songs in each genre (30 seconds per sample).

[0303] Loss function: Classification cross-entropy function.

[0304] Optimizer settings: Select the Adam optimizer, with an initial learning rate of 0.001, and adopt a learning rate decay strategy as training progresses.

[0305] Training epochs setting: Set to 80 epochs based on dataset size and model convergence.

[0306] Batch Size Setting: For training based on two H20 graphics cards, the Batch Size is set to 16.

[0307] Acoustic Scene Inference Unit: By analyzing the reverberation characteristics of the audio (reverberation time RT60, early reflection intensity, etc.) and the ratio of direct sound to reverberant sound, it infers the most likely virtual acoustic environment (such as concert hall, recording studio, livehouse, cathedral) where the music is located.

[0308] The dual-channel stereo audio intelligent mixing to multi-channel immersive sound module is the system's brain, receiving all information from the aforementioned analysis modules (i.e., the AI ​​audio separation module + AI music classification module). Internally, it maintains a sound field rule knowledge base, storing optimal sound field construction rules and mixing strategies preset by professional audio engineers for different music attribute combinations (such as "jazz" + "club environment" + "soothing"). This module performs fusion decisions on the input information (all analyzed information, including acoustic scene, gain, and reverberation), dynamically generating a unique sound field mental model for the current audio. This module defines the fusion decision process for the input information as follows: Global sound field properties: such as overall space size and environmental reverberation characteristics.

[0309] Spatial attributes of each sound source object: Specify a target spatial coordinate and diffusion for each separated sound source object.

[0310] Processing strategy: Specify the desired audio processing type and intensity range for each audio source object.

[0311] Adaptive Multitrack Rendering and Composition Module: This module is the system's execution layer, responsible for the final sound rendering. It contains two processing streams: (1) Substrate acoustic field generation stream (corresponding to the aforementioned substrate acoustic field bitstream); The original stereo signal is copied multiple times to serve as the sound field base. Crucially, this base signal is not directly distributed, but rather, based on the global sound field properties in the sound field psychological model, global environmental reverberation and equalization processing are applied to match it, in order to simulate the target virtual acoustic environment and provide a sense of spatial immersion for the entire sound field.

[0312] (2) Smart object rendering stream (corresponding to the aforementioned object bitstream); For each separated audio source object track, perform the following operations: 1. Model-driven parameter mapping: Based on the spatial attributes and processing strategies specified for the object by the dynamic sound field psychological model, a set of specific collaborative audio processing parameters is mapped.

[0313] 2. Perceptual Optimization Processing Chain: The object track is processed using the aforementioned collaborative audio processing parameters. This processing chain includes at least the following: a. EQ (Electronic Quotient) Shaping: Dynamically adjusts the gain of specific frequency bands based on the target distance and orientation of the sound source. For example, to simulate a far-field sound source, attenuate its high frequencies; to improve the clarity of overhead instruments, enhance their key frequency bands.

[0314] b. Gain control: Adjust the level according to its front and rear depth position.

[0315] c. Delay modulation: Add a micro delay to enhance the sense of positioning and distance.

[0316] d. Object-level reverb: On top of the global reverb, add independent, lightweight, dedicated reverb to important objects (such as lead singers) so that they can both blend into the environment and stand out.

[0317] 3. Spatial rendering: The processed object signal is accurately rendered into the speaker channel corresponding to the target spatial location specified by the sound field psychological model.

[0318] Finally, the signals of each channel of the smart object rendering stream are mixed with the corresponding channel signals of the base sound field generation stream to obtain a 12-channel pseudo-panoramic audio bitstream similar to the 7.1.4 format: left channel L, right channel R, center channel Center, subwoofer channel LFE, left surround channel LS, right surround channel RS, left rear surround channel LB, right rear surround channel RB, front left top channel FLT, front right top channel FRT, rear left top channel RLT, and rear right top channel RRT.

[0319] The in-vehicle pseudo-immersive sound audio mixing and playback processing module compares and calculates the differences between the in-vehicle speaker system and the standard immersive sound playback environment. It then processes the pseudo-immersive sound audio streams with a certain gain ratio and sends them to each speaker channel. Furthermore, considering speaker characteristics, such as the 65-100Hz frequency components of the midrange channel that should be represented by the center speaker, due to the bandwidth limitations of the in-vehicle center speaker, these are distributed to the left and right woofers for low-frequency compensation.

[0320] The content of each channel in the pseudo-immersive sound system is allocated based on the audio channel. To illustrate the logic of the processing method, this application provides a specific example of combining the pseudo-immersive sound audio stream with a 53-speaker system, such as... Figure 5 As shown, the distribution of the 53-unit system is as follows: Front row four-way system (2 super tweeters + 2 tweeters + 2 midrange speakers + 2 woofers); Rear row four-way speaker system (2 super tweeters + 2 tweeters + 2 midrange speakers + 2 woofers). Front center two-way speaker (1 tweeter + 1 midrange speaker); Center-mounted two-way speaker (1 tweeter + 1 midrange speaker); Front row, left and right top-mounted two-way speakers (2 tweeters + 2 midrange speakers); The middle row features a two-way crossover system with two tweeters and two midrange speakers positioned on the top left and right sides. Rear row, left and right top-mounted two-way speakers (2 tweeters + 2 midrange speakers); Rear surround two-way crossover (2 tweeters + 2 midrange speakers); Driver's headrest (left and right, 2 mid-range speakers); Passenger headrest (left and right, 2 mid-range speakers); Left rear headrest (two mid-range speakers); Left and right rear headrest (two mid-range speakers); Driver's seat exciter *2; Passenger seat exciter *2; Left rear seat exciter *2; Right rear seat exciter *2; Heavy bass*1.

[0321] The in-vehicle acoustic acquisition and analysis module uses sensors placed inside the vehicle to acquire acoustic parameters of the interior space, including seat positions, cabin dimensions, sound absorption characteristics of interior materials, and acoustic performance characteristics of each speaker in each passenger position. Combined with subjective listening results, gain coefficients (ga1, ga2, ga3...gan) are calculated to multiply the signal of the current audio object, and these weighted signals are then distributed to the corresponding target speaker channels.

[0322] The speaker channel allocation method based on the pseudo-immersive sound channel is as follows: The left channel bitstream L*ga1 (front left speaker): front left super tweeter, front left tweeter, front left midrange speaker, front left woofer.

[0323] The left channel bitstream L*ga2: rear left super tweeter, rear left tweeter, rear left midrange speaker, rear left woofer, and rear left headrest left and right midrange speakers.

[0324] The right channel bitstream L*ga3: front right super tweeter, front right tweeter, front right midrange speaker, front right woofer.

[0325] The right channel bitstream L*ga4: rear right super tweeter, rear right tweeter, rear right midrange speaker, rear right woofer, and rear right headrest left and right midrange speakers.

[0326] Center channel bitrate Center*ga5: Front center channel two-way (1 tweeter + 1 midrange speaker) Center channel bitrate Center*ga6: Center row center channel two-way (1 tweeter + 1 midrange speaker) Center channel bitrate: Center*ga7: front left woofer, front right woofer.

[0327] Center channel bitrate: Center*ga8: Rear left woofer, rear right woofer.

[0328] The subwoofer channel bitstream LFE*ga9: subwoofer speaker.

[0329] The subwoofer channel bitstream LEF*ga10 contains: driver's seat exciter *2, passenger seat exciter *2, left rear seat exciter *2, and right rear seat exciter *2.

[0330] The left surround channel bitstream LS*ga11: rear left super tweeter, rear left tweeter, rear left midrange speaker, and rear left woofer.

[0331] The left surround channel bitstream LS*ga12: left surround tweeter, left surround midrange speaker.

[0332] The right surround channel bitstream RS*ga13: rear right super tweeter, rear right tweeter, rear right midrange speaker, and rear right woofer.

[0333] The right surround channel bitstream LS*ga14: right surround tweeter, right surround midrange speaker.

[0334] The left rear surround channel bitstream LB*ga15: left surround tweeter, left surround midrange speaker.

[0335] The left rear surround channel bitstream LB*ga16: rear left subwoofer.

[0336] Left rear surround channel bitstream LB*ga17: Driver's headrest left and right mid-range speakers *2.

[0337] The right rear surround channel bitstream RB*gal8: right surround tweeter, right surround midrange speaker.

[0338] The right rear surround channel bitstream LB*ga19: rear right subwoofer.

[0339] The right rear surround channel bitrate is LB*ga20: two mid-range speakers on the left and right sides of the passenger headrest.

[0340] The bitstream of the front left top channel is FLT*ga21: front left top tweeter and midrange speaker. The bitstream of the front left top channel is FLT*ga22: front left subwoofer.

[0341] The bitstream of the front right top channel is FRT*ga23: front left top tweeter and midrange speaker.

[0342] The bitstream of the front right top channel is FRT*ga24: front right subwoofer.

[0343] The bitstream of the rear left top channel is RLT*ga25: rear left top tweeter and midrange speaker.

[0344] The bitstream of the rear left top channel is RRT*ga26: rear left subwoofer.

[0345] The bitstream of the rear right top channel is RRT*ga27: rear right top tweeter and midrange speaker.

[0346] The bitstream of the rear right top channel is RRT*ga28: rear right subwoofer.

[0347] In-vehicle multi-channel audio system and hybrid module: including the whole vehicle speaker system, a tuning system module with multiple functions and interfaces, and power amplifier devices to drive the speakers. This application is not limited to 4, 5, 6, and 7-seater passenger vehicles; the design and description are based on a 5-seater vehicle speaker system as an example.

[0348] The sound field reconstruction calculation module combines the actual vehicle speaker's frequency response, sensitivity, impedance, amplifier's output sensitivity, rated power, and other hardware characteristics, as well as the collected in-vehicle acoustic parameters, to calculate the ideal output signal for each speaker, including volume, delay, phase, and frequency response. The calculated signals allocated to each speaker are then accumulated to generate the final drive signal for that speaker.

[0349] Audio output module: Converts digital signals into analog audio output.

[0350] A multi-functional and interface-based audio mixing system module (i.e., the aforementioned digital signal processing module) can process the audio output from the upmixing module, providing interfaces with various adjustment functions and dimensions. The adjustment modules can be configured through these interfaces, and the audio mixing system module can perform frequency and time domain processing on input music and other audio signals. For example... Figure 6As shown, a 36ch (36-channel) audio tuning system module may include: a preamplifier gain module, a global input gain module, a global EQ adjustment module, a module channel mapping module, a channel replication module, a channel switching module, a single speaker channel EQ adjustment module, a single speaker channel gain adjustment module, a single speaker channel delay module, a single speaker channel phase adjustment module, a single speaker channel mute module, a single speaker channel crossover module, a single speaker channel high-pass filter module, a single speaker channel low-pass filter module, a single speaker channel compression module, a surround algorithm module, a spatial audio algorithm module, a bass enhancement module, a dynamic control module, and a global output gain module. It can be understood that each channel in the 36ch system undergoes processing by the following modules: single speaker channel EQ adjustment module, single speaker channel gain adjustment module, single speaker channel delay module, single speaker channel phase adjustment module, single speaker channel mute module, single speaker channel crossover module, single speaker channel high-pass filter module, single speaker channel low-pass filter module, and single speaker channel compression module.

[0351] A multi-channel car audio system is the carrier and expression of the overall car audio effect. It includes a car infotainment terminal (such as SOC) with audio source playback function, a digital signal processor (DSP), a microcontroller unit (MCU), a power amplifier (power amplifier IC), and various types of speakers connected to the power amplifier (including but not limited to tweeters, midrange speakers, woofers, subwoofers, 3D overhead speakers, 2D surround speakers, and center speakers).

[0352] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These software products are stored in a storage medium and include several instructions to cause an electronic device (which may be a personal computer, server, vehicle, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0353] This application also provides a vehicle including a memory and a processor, the memory storing a computer program that can run on the processor, and the processor executing the computer program to implement any of the methods described above.

[0354] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. The computer-readable storage medium can be transient or non-transient.

[0355] This application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement some or all of the steps in any of the above-described methods. The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0356] It should be noted that, Figure 7 This is a hardware entity diagram of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, the hardware entity of the electronic device 700 includes: a processor 701, a communication interface 702, and a memory 703, wherein: The processor 701 typically controls the overall operation of the electronic device 700.

[0357] Communication interface 702 enables electronic devices to communicate with other terminals or servers via a network.

[0358] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the electronic device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704.

[0359] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0360] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. An audio processing method, characterized in that, The method includes: Determine key information of the original audio data, including multiple sound source objects, category information, and reverberation information. The category information represents the genre to which the original audio data belongs and / or the emotion of the original audio data. Based on the multiple sound source objects, the category information, and the reverberation information, a multi-channel pseudo-immersive sound bitstream is generated; Based on the acoustic parameters corresponding to the current usage scenario and the multi-channel pseudo-immersive sound bitstream, determine the driving signals of multiple target loudspeakers in the current usage scenario; The drive signal for each target loudspeaker is assigned to the corresponding loudspeaker channel for playback; The step of generating a multi-channel pseudo-immersive audio bitstream based on the multiple sound source objects, the category information, and the reverberation information includes: Based on the reverberation information, the category information, and the multiple sound source objects, a sound field effect model is determined; wherein, the sound field effect model includes global sound field attributes, spatial attributes of each sound source object, and processing strategies for each sound source object; based on the global sound field attributes and the original audio data, a base sound field bitstream is generated; based on the spatial attributes of each sound source object and the processing strategies for each sound source object, an object bitstream is generated; based on the base sound field bitstream and the object bitstream, the multi-channel pseudo-immersive sound bitstream is generated.

2. The method according to claim 1, characterized in that, The key information for determining the original audio data includes: Using a pre-trained separation model, the original audio data is separated to obtain the multiple sound source objects; The original audio data is analyzed and processed to obtain the category information and the reverberation information.

3. The method according to claim 2, characterized in that, The separation model includes an encoding module, a separation module, and a decoding module; The process of using a pre-trained separation model to separate the original audio data and obtain the multiple sound source objects includes: The original audio data is preprocessed to obtain the complex spectrum corresponding to the original audio data; The complex spectrum is encoded using the encoding module to obtain the first feature; Using the separation module, the first feature is modeled in the time-frequency domain to obtain the second feature; The decoding module is used to decode the second feature to obtain the plurality of sound source objects.

4. The method according to claim 3, characterized in that, The complex spectrum is used to characterize the frequency range of the speech bands of the original audio data; The encoding process of the complex spectrum to obtain the first feature includes: Using a preset sparse banding strategy, the frequency range of the speech band is divided to obtain multiple first sub-bands; Each of the first sub-bands is subjected to downsampling and compression processing at a corresponding magnification to obtain the first feature.

5. The method according to claim 3, characterized in that, The separation module includes a frequency domain processing unit, a weighting processing unit, and a time domain processing unit; The step of using the separation module to perform time-frequency domain modeling on the first feature to obtain the second feature includes: Using the frequency domain processing unit, the first feature is modeled in frequency band to obtain the third feature; The weighted processing unit is used to perform weighted processing on the first feature and the third feature to obtain the fourth feature; Using the time-domain processing unit, the fourth feature is modeled using frame count to obtain the fifth feature; The weighted processing unit is used to perform weighted processing on the fourth feature and the fifth feature to obtain the second feature.

6. The method according to claim 3, characterized in that, The decoding process of the second feature to obtain the plurality of sound source objects includes: Using a preset sparse banding strategy, the frequency range of the second feature is divided to obtain multiple second sub-bands; Each of the second sub-bands is subjected to upsampling and expansion processing at a corresponding multiple to obtain the plurality of sound source objects.

7. The method according to claim 2, characterized in that, The method further includes: The training samples are input into the separation model to be trained to obtain multiple predicted sound source objects; Based on the predicted multiple sound source objects and the label information corresponding to the training samples, the target loss value is determined; Based on the target loss value, the model parameters of the separation model are updated at least once to obtain the trained separation model.

8. The method according to claim 2, characterized in that, The analysis and processing of the raw audio data to obtain the category information and the reverberation information includes one of the following: The raw audio data is input into a pre-trained first analysis model to obtain the category information and the reverberation information; The original audio data and the multiple sound source objects are input into a pre-trained second analysis model to obtain the category information and the reverberation information.

9. The method according to claim 1, characterized in that, The step of determining the sound field effect model based on the reverberation information, the category information, and the multiple sound source objects includes: Based on the reverberation information, the virtual acoustic environment corresponding to the original audio data is determined; Based on the category information, the multiple sound source objects, and the virtual acoustic environment, the sound field effect model is determined from a preset rule base.

10. The method according to claim 1, characterized in that, The step of generating a base sound field bitstream based on the global sound field attributes and the original audio data includes: Based on the global sound field properties, determine the reverberation processing strategy and the equalization processing strategy; According to the reverb processing strategy, the original audio data is subjected to reverb processing to obtain the first audio data; According to the equalization processing strategy, the first audio data is subjected to equalization processing to obtain the fundamental sound field bitstream.

11. The method according to claim 1, characterized in that, The object bitstream includes the rendering bitstream of the speaker channel corresponding to each of the sound source objects; The process of generating an object bitstream based on the spatial attributes and processing strategy of each sound source object includes: For each sound source object, based on the spatial attributes and processing strategy of the sound source object, the collaborative processing parameters of the sound source object are determined, and based on the collaborative processing parameters, the rendering bitstream of the speaker channel corresponding to the sound source object is generated; wherein, the collaborative processing parameters include at least one of the following: gain parameter, delay parameter, reverberation parameter, and spectrum parameter.

12. The method according to claim 1, characterized in that, The process of generating the multi-channel pseudo-panoramic audio stream based on the base sound field bitstream and the object bitstream includes: The base code stream of each channel in the base sound field code stream and the rendering code stream of each channel in the object code stream are mixed to obtain the multi-channel pseudo-immersive sound code stream; wherein, the multi-channel includes at least three channels selected from the following: left channel, right channel, center channel, subwoofer channel, left surround channel, right surround channel, left rear surround channel, right rear surround channel, front left top channel, front right top channel, rear left top channel, and rear right top channel.

13. The method according to any one of claims 1 to 12, characterized in that, The step of determining the driving signals for multiple target loudspeakers in the current usage scenario based on the acoustic parameters corresponding to the current usage scenario and the multi-channel pseudo-immersive sound bitstream includes: Based on the acoustic parameters corresponding to the current usage scenario, determine the gain coefficient of each sound source object; For each channel in the multi-channel system, the target loudspeaker corresponding to the channel is determined. Based on the gain coefficient of the audio source object corresponding to the channel and the code stream of the channel, the target code stream of the channel is generated, and the target code stream of the channel is used as the driving signal of the target loudspeaker corresponding to the channel.

14. The method according to claim 13, characterized in that, The step of determining the gain coefficient of each sound source object based on the acoustic parameters corresponding to the current usage scenario includes: Based on the acoustic parameters corresponding to the current usage scenario and the hardware parameters of each speaker, determine the ideal signal corresponding to each speaker; For each loudspeaker, based on the ideal signal corresponding to the loudspeaker, the bitstream of the corresponding channel of the loudspeaker is reconstructed to obtain the gain coefficient of the audio source object corresponding to the loudspeaker.

15. The method according to claim 13, characterized in that, The step of generating the target bitstream of the channel based on the gain coefficient of the audio source object corresponding to the channel and the bitstream of the channel includes: The target bitstream of the channel is generated based on the product between the gain coefficient of the audio source object corresponding to the channel and the bitstream of the channel.

16. An audio processing system, characterized in that, The system includes an audio input module, an extraction module, a mixing module, a reconstruction module, and an audio output module, wherein: The audio input module is used to receive raw audio data; The extraction module is connected to the audio input module and is used to determine key information of the original audio data. The key information includes multiple sound source objects, category information and reverberation information. The category information represents the genre to which the original audio data belongs and / or the emotion of the original audio data. The mixing module, connected to the extraction module, is used to generate a multi-channel pseudo-immersive audio stream based on the multiple sound source objects, the category information, and the reverberation information. The reconstruction module, connected to the mixing module, is used to determine the driving signals of multiple target speakers in the current usage scenario based on the acoustic parameters corresponding to the current usage scenario and the multi-channel pseudo-immersive audio bitstream. The audio output module is connected to the reconstruction module and is used to drive each target speaker to play based on the driving signal of each target speaker; The mixing module is used to determine a sound field effect model based on the reverberation information, the category information, and the multiple sound source objects; wherein, the sound field effect model includes global sound field attributes, spatial attributes of each sound source object, and processing strategies for each sound source object; based on the global sound field attributes and the original audio data, a base sound field bitstream is generated; based on the spatial attributes of each sound source object and the processing strategies for each sound source object, an object bitstream is generated; based on the base sound field bitstream and the object bitstream, the multi-channel pseudo-immersive sound bitstream is generated.

17. A vehicle, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program or instructions, which, when executed by the processor, implement the method according to any one of claims 1-15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions, which, when executed by a processor, implement the method described in any one of claims 1-15.

19. A computer program product, characterized in that, The computer program product includes a computer program or instructions, which, when executed by a processor, implement the method described in any one of claims 1-15.

Images