Adaptive sound effect management method and system with scene analysis module

Abstract

The application discloses an adaptive sound effect management method and system with a scene analysis module, relates to the field of adaptive sound effect management, and comprises the following steps: S1, collecting environmental acoustic characteristics and user behavior data in real time through a scene analysis module; S2, dynamically extracting user hearing preference parameters and scene classification labels; S3, constructing a virtual listening sweet spot matched with a physical space; and S4, dynamically adjusting the phase delay and energy distribution ratio of each frequency band to generate an optimal loudness equalization curve meeting multi-scene constraints. Through a multi-level perception-optimization closed-loop architecture, the application realizes adaptive matching of sound field characteristics in a complex dynamic scene, deeply integrates acoustic characteristic extraction and physical space modeling, breaks through the technical boundary of traditional pure signal processing, and solves the multi-objective conflict problem based on cross-modal collaborative learning and a global optimization algorithm.

Description

Technical Field

[0001] This invention belongs to the field of adaptive sound effect management, and more specifically, relates to an adaptive sound effect management method with a scene analysis module. Furthermore, this invention also relates to an adaptive sound effect management system with a scene analysis module. Background Technology

[0002] When using wireless devices, it is often necessary to adjust the device's sound effects according to changes in the environment, which is constantly changing. When a user moves from a quiet place to a noisy place, in order to obtain better sound effects and improve voice clarity, the user often needs to adjust the volume according to the environment.

[0003] For example, the method and device for adaptive sound effect adjustment disclosed in patent CN102436821A employs an environment adaptation strategy based on a fixed acoustic model in its adaptive sound effect management system. Its core defects are reflected in the following aspects:

[0004] The environmental perception dimension is singular, relying on data acquisition from a single microphone array or a limited number of sensors, and fails to achieve joint modeling of acoustic features and spatial geometric constraints, resulting in the sound field reconstruction process ignoring obstacle reflections and dynamic factors of user movement trajectory;

[0005] The feature extraction mechanism is rigid, and the frequency feature extraction method does not combine the deep correlation of multimodal behavioral data. In particular, in dynamic scene classification, it lacks collaborative analysis of user visual attention and body movements, resulting in a mismatch between scene labels and auditory needs.

[0006] The sound field optimization strategy is fragmented, employing passive reverberation suppression or beamforming as independent operating modes, without establishing a global optimization framework for sound field synthesis and speaker distribution;

[0007] Based on the above, we propose an adaptive sound effect management method and system with a scene analysis module, which effectively addresses the aforementioned shortcomings. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by proposing an adaptive sound effect management method and system with a scene analysis module. Through a multi-level perception-optimization closed-loop architecture, it achieves adaptive matching of sound field characteristics in complex dynamic scenes, deeply integrates acoustic feature extraction and physical space modeling, breaks through the technical boundaries of traditional pure signal processing, and solves the multi-objective conflict problem based on cross-modal collaborative learning and global optimization algorithms.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] An adaptive sound effect management method with a scene analysis module includes the following steps:

[0011] S1. Real-time collection of environmental acoustic features and user behavior data through the scene analysis module. The environmental acoustic features include spatial reflection parameters, background noise spectrum and sound source distribution characteristics.

[0012] S2. Based on a deep neural network fusion of multimodal sensors, dynamically extract user listening preference parameters and scene classification labels. The listening preference parameters include frequency response sensitivity, dynamic range threshold and sound field diffusion preference.

[0013] S3. Based on the user's real-time location coordinates and head orientation angle, call the sound field reconstruction algorithm to optimize the sound wave propagation path distribution and construct a virtual listening sweet zone that matches the physical space.

[0014] S4. By using a programmable digital filter bank and an adaptive gain controller, combined with the classification labels generated in step S2 and the reconstruction parameters in step S3, the phase delay and energy distribution ratio of each frequency band are dynamically adjusted to generate the optimal loudness equalization curve that meets the constraints of multiple scenarios.

[0015] Preferably, the environmental acoustic feature acquisition in step S1 further includes: integrating the azimuth data of the inertial measurement unit and the obstacle distance information of the millimeter-wave radar through a multimodal sensor fusion algorithm to establish a three-dimensional sound wave propagation obstacle model; specifically including:

[0016] S11. The particle filter algorithm is used to fuse the pose data of six degrees of freedom from the IMU to generate a real-time probability distribution map of the user's head movement.

[0017] S12. Construct the reflection coefficient matrix of spatial obstacles using the flight time measurement values ​​of millimeter-wave radar, and calculate the path attenuation factor of direct sound and reflected sound based on the geometric acoustic model.

[0018] S13. Combining the output parameters of steps S11 and S12, the Markov decision process is applied to dynamically predict the optimal diffraction path of the sound wave in a complex obstacle environment, thereby reducing multipath interference and increasing the proportion of direct sound energy.

[0019] Preferably, the deep neural network in step S2 includes an adversarial training mechanism, specifically:

[0020] The generator network uses a bidirectional LSTM structure to synthesize personalized preference feature vectors based on historical listening data;

[0021] The discriminator network adopts a residual convolutional architecture and uses the joint analysis of spectrograms and HRTF parameters to determine the authenticity of generated data.

[0022] The design incorporates a dynamic weighted adaptive loss function, which reduces the user preference prediction error of the model in scenarios with limited samples.

[0023] Preferably, the virtual listening sweet zone construction process in step S3 includes a dynamic compensation mechanism, the specific steps of which are as follows:

[0024] S31. Calculate the individual differences in ear canal resonance frequency parameters using a bioacoustic model and establish a personalized frequency response equalization curve;

[0025] S32. Combine user pupil tracking data with EEG alpha band energy values ​​to assess auditory attention concentration in real time.

[0026] S33. When attention drift is detected to exceed the threshold, the comb filter effect in the 10-14kHz frequency band is automatically enhanced to improve sound image localization perception.

[0027] Preferably, the generation of the optimal loudness equalization curve in step S4 also includes a multi-device collaborative optimization process, specifically:

[0028] S41. Synchronize the phase difference of distributed speakers through the latency compensation protocol of Bluetooth Mesh network;

[0029] S42. Based on the Nash equilibrium theory of game theory, establish a power distribution strategy for each loudspeaker unit to ensure that the sound pressure level uniformity reaches within ±1.5dB.

[0030] S43. The acoustic fingerprint matching algorithm is used to eliminate the color distortion caused by frequency response differences between devices, and the collaborative optimization improves the sound field coherence of the multi-channel system.

[0031] Preferably, the deep neural network in step S2 employs a cross-modal attention mechanism, specifically including:

[0032] The audio branch network extracts frequency domain feature maps through three layers of dilated convolutions;

[0033] The visual branch network extracts scene semantic embedding vectors through a two-stream TSN model;

[0034] The cross-modal collaborative module implements spectral domain attention weight allocation, and its attention coefficient calculation formula is as follows:

[0035] ,in, For the first Each audio feature unit For the first Each visual feature unit For trainable projection matrices, Attention coefficient The above formula is used to calculate the activation function, thereby improving the accuracy of complex scene classification.

[0036] Preferably, the sound field reconstruction algorithm in step S3 includes a beamforming and sound field synthesis co-optimization process, specifically including:

[0037] S31. Generate a personalized head-related transfer function based on the user's auricular HRTF model;

[0038] S32. Calculate the optimal beam weight vector using the minimum variance distortion-free response algorithm:

[0039] ,in, The optimal beam weight vector is used to achieve directional control of the sound beam in space by complex weighting of the output signals of each loudspeaker unit; The noise covariance matrix is ​​obtained by... The computation achieves inverse compensation of environmental noise characteristics, improving the signal-to-noise ratio; The desired direction array manifold vector describes the spatial transfer characteristics of the loudspeaker array in the θ direction. After being coupled with the HRTF parameters, it generates the array-ear joint transfer function.

[0040] S33. Combining HRTF parameters and real-time user coordinates, the boundary conditions of the Helmholtz equation are iteratively solved in the near-field sound field model. The hybrid iterative algorithm used in the near-field sound field model is as follows:

[0041] By deeply coupling the real-time inverse operation of the noise covariance matrix with the HRTF parameters, we can achieve environmental noise adaptive beamforming and precise near-field sound image localization in the three-dimensional audio field.

[0042] An adaptive sound effects management system with a scene analysis module is used to implement the above methods, including:

[0043] The scene analysis module consists of a distributed microphone array, a ToF lidar and an inertial measurement unit. It is used to collect environmental acoustic features and user behavior data in real time, and output spatial reflection parameters, background noise spectrum and real-time user location coordinates.

[0044] The feature extraction module integrates a deep neural network processor with a multimodal sensor fusion interface, receives the output data from the scene analysis module, and generates auditory preference parameters and scene classification labels;

[0045] The sound field optimization module is equipped with an FPGA logic unit that is the core of the sound field reconstruction algorithm. It dynamically optimizes the sound wave propagation path based on the user's head orientation angle and position coordinates, and generates boundary constraints for the virtual listening sweet zone.

[0046] The sound effect generation module includes a DSP processing core with a programmable digital filter bank and a dynamic gain distribution circuit. It receives the classification labels from the feature extraction module and the boundary parameters from the sound field optimization module, and outputs the optimal audio drive signal that adjusts the phase delay and energy distribution.

[0047] The scene analysis module is connected to the feature extraction module via the PCIe bus, and the sound field optimization module receives the intermediate layer feature tensor of the feature extraction module through a high-speed parallel interface and transmits the reconstruction parameters to the sound effect generation module through the AES / EBU digital audio interface.

[0048] Preferably, the scene analysis module includes the following sub-units:

[0049] The acoustic sensor array consists of a 32-channel ring microphone array, which extracts multipath reflection time series using a beamforming algorithm.

[0050] The lidar array uses multi-wavelength ToF sensors to construct a reflection coefficient matrix of spatial obstacles;

[0051] Data fusion unit: Aligns the data from the acoustic sensor group with the point cloud information from the lidar group using the timestamps of the acoustic sensor group to generate a three-dimensional sound field propagation model with sub-microsecond synchronization accuracy;

[0052] The data from the acoustic sensor group and the lidar group are input into the data fusion unit through the LVDS interface, and the fusion result is transmitted to the feature extraction module via gigabit Ethernet protocol.

[0053] Technical effects and advantages of the present invention: Compared with the prior art, the adaptive sound effect management method and system with scene analysis module provided by the present invention have the following advantages:

[0054] The deep fusion of multi-dimensional environmental perception, through the multimodal data fusion of distributed microphone array, ToF lidar and inertial measurement unit, realizes the geometric constraint modeling of sound wave propagation path and the physical modeling of dynamic obstacle reflection characteristics, fundamentally solving the problem of limited dimensions of environmental parameter acquisition in traditional methods;

[0055] Cross-modal feature collaborative learning, by leveraging the adversarial training mechanism of deep neural networks and cross-modal attention models, simultaneously explores the deep correlation between users' listening preferences and behavioral characteristics, breaking through the cognitive boundaries of traditional solutions that independently analyze audio features, and significantly improving the semantic consistency of scene classification.

[0056] Global optimization of sound field synthesis, based on hybrid optimization theory, such as the joint solution of Nash equalization and boundary element method, constructs an integrated model of sound field reconstruction and equipment coordination, which achieves optimal energy distribution under multi-objective constraints while ensuring the accuracy of three-dimensional sound image positioning, and overcomes the inherent defects of local optimization in traditional methods;

[0057] The low-latency, high-efficiency architecture design, through triple optimization of hierarchical clock synchronization mechanism, high-speed bus protocol between modules and dynamic power management strategy, ensures the real-time response capability of the system in complex scenarios, while breaking through the energy efficiency bottleneck of traditional architecture and achieving compatibility between professional-grade sound quality and mobile device battery life. Attached Figure Description

[0058] Figure 1 This is a flowchart of the adaptive sound effect management method with a scene analysis module according to the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0060] This invention provides an adaptive sound effect management method and system with a scene analysis module. Through a multi-layered perception-optimization closed-loop architecture, it achieves adaptive matching of sound field characteristics in complex dynamic scenes.

[0061] This solution deeply integrates acoustic feature extraction with physical space modeling, breaking through the technical boundaries of traditional pure signal processing. Based on cross-modal collaborative learning and global optimization algorithms, it solves the problem of multi-objective conflict. Through hardware-algorithm co-design, it ensures the system's comprehensive advantages in real-time performance and power consumption control. This solution provides a general sound field management paradigm for dynamic scenarios in the field of intelligent audio, possessing both theoretical universality and engineering application value.

[0062] like Figure 1 As shown, the adaptive sound effect management method with a scene analysis module includes the following steps:

[0063] S1. Real-time acquisition of environmental acoustic features and user behavior data through a scene analysis module. The environmental acoustic features include spatial reflection parameters, background noise spectrum, and sound source distribution characteristics. The environmental acoustic feature acquisition in step S1 further includes: integrating the azimuth data of the inertial measurement unit (IMU) with obstacle distance information from millimeter-wave radar using a multimodal sensor fusion algorithm to establish a three-dimensional sound wave propagation obstacle model. Specifically, this includes:

[0064] a) A particle filter algorithm is used to fuse six degrees of freedom pose data from the IMU to generate a real-time probability distribution map of the user's head motion;

[0065] b) Construct the reflection coefficient matrix of spatial obstacles using time-of-flight measurements from millimeter-wave radar, and calculate the path attenuation factors of direct and reflected sound based on a geometric acoustic model;

[0066] c) Combining the output parameters of steps a) and b), Markov decision process is applied to dynamically predict the optimal diffraction path of sound waves in complex obstacle environments;

[0067] In step S1, the scene analysis module uses a distributed microphone array and a ToF lidar to work together. When calculating spatial reflection parameters through the spatiotemporal correlation of multi-channel signals, it specifically includes: establishing the geometric constraint equations for the sound wave propagation path.

[0068] , where c ij Let represent the coupling coefficient between the i-th microphone and the j-th reflector. For microphone spatial coordinates, Let k be the three-dimensional coordinates of the sound source. For the corresponding frequency of sound wave wavelength, The angle of incidence of the sound source. The calculation method reduces the modeling error of spatial reflection characteristics by taking the time delay of the sound wave reaching the microphone after passing through the j-th reflecting surface.

[0069] S2. Based on a deep neural network fusion of multimodal sensors, dynamically extract user listening preference parameters and scene classification labels. The listening preference parameters include frequency response sensitivity, dynamic range threshold, and sound field diffusion preference. The deep neural network in step S2 includes an adversarial training mechanism, specifically implemented as follows:

[0070] The generator network uses a bidirectional LSTM structure to synthesize personalized preference feature vectors based on historical listening data;

[0071] The discriminator network adopts a residual convolutional architecture and uses the joint analysis of spectrograms and HRTF parameters to determine the authenticity of generated data.

[0072] The design incorporates a dynamically weighted adaptive loss function:

[0073] ;

[0074] in, p represents the expected discriminant value of a real sample, where x is a sample from the real auditory dataset (including spectrograms and HRTF parameters). data The data distribution represents the real data; it is used to drive the discriminator to improve its ability to recognize real user preference data and ensure that the feature vector output by the generator conforms to real physiological auditory characteristics.

[0075] z is the latent variable input that follows a Gaussian distribution pz, and G(z) is the preference feature vector synthesized by the generator. By minimizing the discrimination score of the generated data, the generator is prompted to learn the statistical characteristics of the latent space of user preferences, thus breaking through the dependence of traditional supervised learning on labeled data.

[0076] , The samples are random points on the line connecting the real data and the generated data. D( ) for the discriminator in The gradient at the point forces the Lipschitz constant of the discriminator to be close to 1, avoiding training collapse and solving the collapse problem of traditional GAN ​​mode.

[0077] Through the above, modal compatibility is enhanced: the introduction of the gradient penalty term enables the discriminator to maintain stable training dynamics in a high-dimensional mixed data space of spectral time-frequency characteristics and HRTF spatial parameters, avoiding feature confusion caused by data distribution shift.

[0078] Breakthrough in small sample generalization ability: Through the latent variable generation mechanism and adversarial game framework, the deep correlation of limited user behavior data is effectively mined to generate pseudo samples that conform to personalized listening perception distribution, which significantly improves the prediction accuracy in cold start scenarios.

[0079] Physical constraint embedding: The discriminator's joint discrimination operation on HRTF parameters hard-codes the physiological characteristics of the human auditory organ into the adversarial training process, ensuring that the generated feature vector conforms to the physical laws of sound wave propagation;

[0080] Training efficiency optimization: The dynamic weight coefficient λ is adjusted adaptively based on the discriminator gradient magnitude, which improves the network convergence speed and eliminates the need for manual parameter tuning.

[0081] It should be noted that the deep neural network in step S2 employs a cross-modal attention mechanism, specifically including:

[0082] The audio branch network extracts frequency domain feature maps through three layers of dilated convolutions;

[0083] The visual branch network extracts scene semantic embedding vectors through a two-stream TSN model;

[0084] The cross-modal collaborative module implements spectral domain attention weight allocation, and its attention coefficient calculation formula is as follows:

[0085] ,in, For the first Each audio feature unit For the first Each visual feature unit For trainable projection matrices, Attention coefficient The above formula is used to calculate the activation function, thereby improving the accuracy of complex scene classification.

[0086] S3. Based on the user's real-time location coordinates and head orientation angle, the sound field reconstruction algorithm is invoked to optimize the sound wave propagation path distribution and construct a virtual listening sweet zone that matches the physical space; the virtual listening sweet zone construction process in step S3 includes a dynamic compensation mechanism, and the specific implementation steps are as follows:

[0087] 1) Calculate individual differences in ear canal resonant frequencies using bioacoustic models to establish personalized frequency response equalization curves;

[0088] 2) Combine user pupil tracking data with EEG alpha band energy values ​​to assess auditory attention concentration in real time;

[0089] 3) When attention drift is detected to exceed the threshold, the comb filtering effect in the 10-14kHz frequency band is automatically enhanced to improve sound image localization perception; the compensation mechanism extends the duration of the immersive audio experience by 2.3 times.

[0090] Furthermore, the sound field reconstruction algorithm in step S3 includes a beamforming and sound field synthesis co-optimization process, specifically including:

[0091] S31. Generate a personalized head-related transfer function based on the user's auricular HRTF model;

[0092] S32. Calculate the optimal beam weight vector using the minimum variance distortion-free response algorithm:

[0093] ,in, The optimal beam weight vector is used to achieve directional control of the sound beam in space by complex weighting of the output signals of each loudspeaker unit; The noise covariance matrix is ​​obtained by... The computation achieves inverse compensation of environmental noise characteristics, improving the signal-to-noise ratio; The desired direction array manifold vector describes the spatial transfer characteristics of the loudspeaker array in the θ direction. After being coupled with the HRTF parameters, it generates the array-ear joint transfer function.

[0094] S33. Combining HRTF parameters and real-time user coordinates, the boundary conditions of the Helmholtz equation are iteratively solved in the near-field sound field model. The hybrid iterative algorithm used in the near-field sound field model is as follows:

[0095] By deeply coupling the real-time inverse operation of the noise covariance matrix with the HRTF parameters, we can achieve environmental noise adaptive beamforming and precise near-field sound image localization in the three-dimensional audio field.

[0096] In addition, the sound field propagation path optimization adopts an adaptive waveguide model, the core calculation equation of which is:

[0097] ,in, The effective sound field propagation path length, This represents the amplitude attenuation factor for the i-th reflection path, where k is the wavenumber. Let be the Euclidean distance from the user's position to the i-th speaker. As a phase compensation term, this formula enhances direct sound through constructive interference of the waveguide path, thereby improving speech intelligibility in a reverberant environment and reducing the comb filtering effect to an imperceptible level.

[0098] S4. Using a programmable digital filter bank and an adaptive gain controller, combined with the classification labels generated in step S2 and the reconstruction parameters in step S3, the phase delay and energy distribution ratio of each frequency band are dynamically adjusted to generate an optimal loudness equalization curve that satisfies multiple scenario constraints; the generation of the optimal loudness equalization curve in step S4 also includes a multi-device collaborative optimization process, specifically implemented as follows:

[0099] a) Synchronize the phase difference of distributed speakers using the latency compensation protocol of Bluetooth Mesh network;

[0100] b) Establish a power allocation strategy for each loudspeaker unit based on the Nash equilibrium theory of game theory to ensure that the sound pressure level uniformity is within ±1.5dB;

[0101] c) Use an acoustic fingerprint matching algorithm to eliminate hue distortion caused by frequency response differences between devices.

[0102] In addition, the sound field reconstruction algorithm includes a multi-user collaborative processing mechanism, the specific implementation of which is as follows:

[0103] 1) Real-time tracking of the coordinates and movement speed of multiple users through an ultra-wideband positioning system;

[0104] 2) Apply the Voronoi diagram segmentation algorithm to divide the priority radiation area for each user;

[0105] 3) Design a time-domain staggered beam focusing strategy to concentrate the acoustic energy within a 30-100ms time window to the auditory perception area of ​​high-priority users; this mechanism improves the individual speech recognition in multi-user concurrent scenarios.

[0106] The digital filter bank in step S4 adopts a dynamic coefficient update strategy, and its parameter adjustment rules are as follows:

[0107] ,in, Here, α is the gain compensation value required for the m-th critical frequency band, and α is the frequency response matching weighting coefficient. To optimize the target transfer function to conform to international electroacoustic standards. This is the environment transfer function measured through the scenario analysis module. / Let be the lower / upper cutoff frequency of the m-th frequency band, and β be the noise robustness adjustment factor, whose value is positively correlated with the room reverberation time. Let m be the signal-to-noise ratio of the m-th frequency band;

[0108] The above formula, through a weighted combination of frequency response matching and noise suppression, simultaneously addresses the following issues:

[0109] Frequency domain distortion correction: Generates accurate compensation curves for environmental acoustic defects;

[0110] Temporal noise suppression: Suppresses reverberant tails and background noise in low signal-to-noise ratio regions;

[0111] Perceptual consistency preservation: based on equal loudness curve constraints The allocation strategy ensures that the compensation result conforms to the characteristics of human hearing.

[0112] This embodiment also proposes an adaptive sound effect management system with a scene analysis module. This system is used to implement the above-mentioned method, including:

[0113] Scene analysis module: Composed of a distributed microphone array, ToF lidar, and inertial measurement unit, it is used to collect environmental acoustic characteristics and user behavior data in real time, and output spatial reflection parameters, background noise spectrum, and real-time user location coordinates; the scene analysis module includes the following sub-units:

[0114] Acoustic sensor array: Consists of a 32-channel ring microphone array with a sampling rate of 192kHz, and extracts multipath reflection time series through beamforming algorithm;

[0115] LiDAR group: Employs multi-wavelength ToF sensors with a detection accuracy of ±5mm, constructing a reflection coefficient matrix of spatial obstacles;

[0116] Data fusion unit: By fusing the timestamp-aligned data of the acoustic sensor group and the point cloud information of the lidar group through an extended Kalman filter, a three-dimensional sound field propagation model with sub-microsecond synchronization accuracy is generated, and the error rate of the model is less than 2.3%; wherein, the data of the acoustic sensor group and the lidar group are input into the data fusion unit through the LVDS interface, and the fusion result is transmitted to the feature extraction module via gigabit Ethernet protocol.

[0117] Feature extraction module: A deep neural network processor integrating a multimodal sensor fusion interface, which receives the output data from the scene analysis module and generates auditory preference parameters and scene classification labels; the feature extraction module includes the following processing links:

[0118] Audio feature extraction link: The input acoustic signal is divided into 1 / 3 octave bands using a third-order anti-aliasing FIR filter bank, and the Mel-frequency cepstral coefficients and dynamic range compression parameters are extracted through time-frequency transformation;

[0119] Behavior-aware link: Deploy a dual-branch graph convolutional network to process six-axis motion data from the IMU and visual attention heatmaps from the pupil-tracking sensor, respectively;

[0120] Cross-modal fusion unit: Applies a gated attention mechanism to weighted aggregate audio and behavioral features. The attention weights are calculated as follows:

[0121] ,in, Let i be the i-th audio feature vector. Let j be the feature vector of the j-th behavior. The gating weight matrix is ​​used; the processing link achieves zero-latency interaction of multimodal features through a shared memory architecture, thereby reducing the overall inference latency.

[0122] Sound field optimization module: An FPGA logic unit equipped with a core sound field reconstruction algorithm dynamically optimizes the sound wave propagation path based on the user's head orientation angle and position coordinates, generating boundary constraints for the virtual listening sweet spot; the architecture of the sound field optimization module includes:

[0123] Personalized HRTF database: Stores more than 500 sets of head-related transfer function measurement data, each dataset containing impulse responses at a 360° spatial azimuth angle at a 48kHz sampling rate;

[0124] Real-time sound field solver: Based on a CUDA-accelerated boundary element method engine, it dynamically calculates the sound pressure distribution at the user's ear position. The calculation expression is as follows: In the formula, Let be the equivalent strength of the nth virtual source. Let k be the virtual source coordinates and k be the sound wavenumber. is the scaling factor of the three-dimensional free-field Green's function, used to correct for the natural attenuation caused by the diffusion of spherical waves. Its reciprocal relationship ensures the dimensional consistency between the far-field and near-field models.

[0125] Adaptive beam controller: The hybrid MUSIC algorithm is used to calculate the optimal loudspeaker excitation weight parameters and the calibration parameters are written into the register of the power amplifier module via the I²C bus. The control cycle is accurate to the 1ms level.

[0126] The above formula combines a personalized HRTF database with virtual source reconstruction technology, achieving two major breakthroughs in the sound field optimization module: accurate sound image modeling: through virtual source equivalent intensity Spatial coordinates Dynamic matching accurately simulates the propagation path of sound waves after reflection and diffraction through complex obstacles, making the three-dimensional sound pressure distribution of the virtual listening sweet spot highly consistent with the characteristics of real physical space.

[0127] Real-time computing performance: complex phase term The GPU parallel computing architecture, combined with the CUDA accelerated boundary element method engine, meets the real-time requirement of 1ms control cycle and breaks through the traditional acoustic simulation computing power bottleneck.

[0128] Multi-physics coupling: The frequency-varying characteristics of wavenumber k and the dynamic adjustment of the digital filter bank form a cross-domain coupling, realizing a quantitative improvement in the speech intelligibility index, while also using scaling factors. Ensure numerical stability when switching between far-field and near-field models.

[0129] Adaptive Beam Controller: This module employs a hybrid MUSIC algorithm to calculate the optimal speaker excitation weight parameters and writes the calibration parameters into the power amplifier module's registers via the I²C bus. The control cycle is accurate to the 1ms level. This module achieves high-performance beam manipulation through the following technological improvements:

[0130] Multi-resolution spectral estimation architecture:

[0131] The problem of azimuth ambiguity of broadband sound sources is decomposed by front / back spatial smoothing based on hybrid MUSIC algorithm;

[0132] The inter-channel coupling effect of distributed loudspeaker arrays is eliminated by combining covariance matrix reconstruction technology;

[0133] Dynamically adjust the spatial spectral peak search range to adapt to the location changes of mobile users in real time;

[0134] Calibration parameter writing mechanism:

[0135] The power amplifier module is configured with a dedicated register set, including a divider register, a gain adjustment register, and a delay compensation register;

[0136] Microsecond-level parameter synchronization is achieved through the multi-master arbitration protocol of the I²C bus, and redundancy checks are implemented to prevent signal distortion.

[0137] It supports burst transmission mode, and can complete batch updates of parameters for 128 speaker nodes in a single communication cycle;

[0138] Precision timing management:

[0139] It adopts a hierarchical clock tree architecture, with the main control FPGA outputting a nanosecond-precise trigger pulse sequence;

[0140] An integrated dynamic phase-locked loop module eliminates accumulated phase errors between multiple clock domains, ensuring the stability of the 1ms control cycle.

[0141] Audio Effects Generation Module: This module comprises a DSP processing core with a programmable digital filter bank and dynamic gain allocation circuitry. It receives classification labels from the feature extraction module and boundary parameters from the sound field optimization module, and outputs an optimal audio drive signal that adjusts phase delay and energy distribution. This module achieves multi-objective optimization through the following core technologies:

[0142] Filter bank dynamic configuration mechanism:

[0143] The multiphase filter bank design supports 1 / 24 octave band subdivision and can dynamically load 128 sets of time-varying impulse response parameters; the coefficients of each sub-band filter are reconstructed in real time according to the boundary parameters of the sound field optimization module, giving priority to compensating for the user's sensitive frequency bands; the stopband attenuation depth is set in combination with the auditory masking effect model to avoid auditory fatigue from high-frequency harmonic components.

[0144] Gain allocation strategy:

[0145] Based on a dual-threshold dynamic compression strategy, the energy allocation rules are distinguished between the environmental noise-dominated area and the active speech signal area; an asymmetric limiting function is introduced to prevent transient sound signal overload while preserving the dynamic fluctuation characteristics of the music signal; an energy prediction model is established by combining historical scene tags, and gain margins for each frequency band are pre-allocated to reduce adjustment delay.

[0146] Cross-domain collaborative optimization:

[0147] The sound field reconstruction parameters are received through the AES / EBU interface, and the phase delay is pre-compensated in the time domain and interpolated in the frequency domain. A multi-layer pipeline architecture is adopted to split the signal processing chain into three parallel computing stages: frame synchronization, frequency band cutting, and dynamic equalization. An on-chip error correction unit is integrated, and BCH encoding is used to ensure the transmission integrity of the distributed speaker drive signal.

[0148] The scene analysis module is connected to the feature extraction module via a PCIe bus, and the sound field optimization module receives the intermediate layer feature tensor from the feature extraction module through a high-speed parallel interface and transmits the reconstruction parameters to the sound effect generation module through the AES / EBU digital audio interface.

[0149] The module synergy effect of the above system is as follows:

[0150] Real-time performance guarantee: The parameter update rate of the beam controller and the frame processing cycle of the audio module are precisely aligned through timestamps;

[0151] Resource reuse mechanism: The two share the scene classification labels of the feature extraction module and reuse the azimuth index data of the HRTF database;

[0152] Multi-objective optimization: The spatial acoustic energy focusing of beamforming and the frequency domain energy allocation of sound effect generation form a closed loop for solving the joint cost function;

[0153] Hardware architecture adaptation: High parallelism FPGA and ultra-low latency DSP chip are used respectively to match the characteristics of their respective computationally intensive and control-intensive tasks.

[0154] In summary, the present invention has the following effects:

[0155] Deep fusion of multi-dimensional environmental perception: By fusing multi-modal data from a distributed microphone array, ToF lidar, and inertial measurement unit, geometric constraint modeling of sound wave propagation paths and physical modeling of dynamic obstacle reflection characteristics are achieved, fundamentally solving the problem of limited dimensions of environmental parameter acquisition in traditional methods;

[0156] Cross-modal feature collaborative learning: By leveraging the adversarial training mechanism of deep neural networks and cross-modal attention models, we can simultaneously explore the deep correlation between users' listening preferences and behavioral features, break through the cognitive boundaries of traditional solutions that independently analyze audio features, and significantly improve the semantic consistency of scene classification.

[0157] Global optimization sound field synthesis: Based on hybrid optimization theory, such as the joint solution of Nash equalization and boundary element method, an integrated model of sound field reconstruction and equipment coordination is constructed. While ensuring the accuracy of three-dimensional sound image positioning, the optimal energy distribution under multi-objective constraints is achieved, overcoming the inherent defects of local optimization in traditional methods.

[0158] Low-latency and high-efficiency architecture design: Through triple optimization of hierarchical clock synchronization mechanism, high-speed bus protocol between modules and dynamic power management strategy, the system ensures real-time response capability in complex scenarios, while breaking through the energy efficiency bottleneck of traditional architecture and achieving compatibility between professional-grade sound quality and mobile device battery life.

[0159] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method of adaptive sound effect management with a scene analysis module, characterized in that, Comprising the following steps: S1, collecting environmental acoustic characteristics and user behavior data in real time through a scene analysis module, the environmental acoustic characteristics including spatial reflection parameters, background noise spectrum, and sound source distribution characteristics; S2, dynamically extracting user hearing preference parameters and scene classification labels based on a deep neural network with multi-modal sensor fusion, the hearing preference parameters including frequency response sensitivity, dynamic range threshold, and sound field diffusion preference; S3, calling a sound field reconstruction algorithm to optimize sound wave propagation path distribution according to user real-time position coordinates and head orientation angle, and constructing a virtual listening sweet spot matching the physical space; the sound field reconstruction algorithm in step S3 includes a beamforming and sound field synthesis collaborative optimization process, specifically comprising: S31, generating an individualized head-related transfer function according to the user's pinna HRTF model; S32, calculating the optimal beam weight vector through the minimum variance distortionless response algorithm: wherein, is the optimal beam weight vector, realizing directional regulation of sound beams in space by complex weighting of each loudspeaker unit output signal; is the noise covariance matrix, realizing inverse compensation of environmental noise characteristics by operation, improving the signal-to-noise ratio; is the expected direction array manifold vector, describing the spatial transfer characteristics of the loudspeaker array in the θ direction, and after coupling with the HRTF parameter, the array-human ear joint transfer function is generated; S33, combining HRTF parameters and user real-time coordinates to iteratively solve the boundary conditions of the Helmholtz equation in the near-field sound field model, and using a hybrid iterative algorithm in the near-field sound field model; Through real-time inverse operation of the noise covariance matrix and deep coupling of the HRTF parameters, adaptive beamforming of environmental noise and accurate positioning of the near-field sound image are realized in the field of three-dimensional audio; S4, dynamically adjusting the phase delay and energy distribution ratio of each frequency band through a programmable digital filter bank and an adaptive gain controller to generate an optimal loudness equalization curve that meets multi-scene constraints.

2. The adaptive sound management method with a scene analysis module according to claim 1, characterized in that, The environmental acoustic characteristic collection in step S1 further comprises: integrating the orientation data of the inertial measurement unit and the obstacle distance information of the millimeter wave radar through a multi-modal sensor fusion algorithm to establish a three-dimensional sound wave propagation obstacle model; specifically comprising: S11, using a particle filter algorithm to fuse six-degree-of-freedom pose data from the IMU to generate a real-time probability distribution map of user head movement; S12, constructing a reflection coefficient matrix of spatial obstacles through the time-of-flight measurement value of the millimeter wave radar, and calculating the path attenuation factor of direct sound and reflected sound based on a geometric acoustics model; S13, combining the output parameters of steps S11 and S12, and applying a Markov decision process to dynamically predict the best diffraction path of sound waves in a complex obstacle environment, thereby reducing multipath interference and increasing the proportion of direct sound energy.

3. The adaptive sound management method with a scene analysis module according to claim 1, characterized in that, The deep neural network in step S2 includes an adversarial training mechanism, specifically: The generator network uses a bidirectional LSTM structure to synthesize individualized preference feature vectors based on historical hearing data; The discriminator network uses a residual convolution architecture to generate data authenticity through joint discrimination of frequency spectrum and HRTF parameters; A dynamic weight adaptive loss function is designed to reduce user preference prediction errors in limited sample scenarios.

4. The adaptive sound management method with a scene analysis module according to claim 1, characterized in that, The virtual listening sweet spot construction process in step S3 includes a dynamic compensation mechanism, specifically comprising: S31, calculating individual difference parameters of ear canal resonance frequency through a bioacoustic model to establish an individualized frequency response equalization curve; S32, combining user pupil tracking data and alpha band energy values of brain waves to evaluate the auditory attention concentration index in real time; S33、When the attention drift is detected to exceed the threshold, the comb filter effect of the 10-14 kHz frequency band is automatically enhanced to improve the sound image localization perception.

5. The adaptive sound management method with a scene analysis module according to claim 1, characterized in that, The optimal loudness equalization curve generation in step S4 further includes a multi-device collaborative optimization process, specifically: S41, synchronize the phase difference of the distributed loudspeakers through the time delay compensation protocol of the Bluetooth Mesh network; S42, establish a power distribution strategy for each loudspeaker unit based on the Nash equilibrium theory of game theory, to ensure that the sound pressure level uniformity is within ±1.5dB; S43, use the acoustic fingerprint matching algorithm to eliminate the color distortion caused by the frequency response difference between devices, and cooperatively optimize the coherence of the multi-channel system sound field.

6. The adaptive sound management method with a scene analysis module according to claim 3, characterized in that, The deep neural network in step S2 adopts a cross-modal attention mechanism, specifically including: The audio branch network extracts a frequency domain feature map through three layers of dilated convolution; The visual branch network extracts a scene semantic embedding vector through a dual-flow TSN model; The cross-modal collaborative module implements spectral domain attention weight distribution, and its attention coefficient calculation formula is: wherein, is the audio feature unit, is the visual feature unit, is the trainable projection matrix, is the attention coefficient, is the activation function; the computer mechanism through the above formula makes the complex scene classification accuracy rate improve.

7. An adaptive sound management system with a scene analysis module, characterized in that The system is used to implement the method of any one of claims 1-6, comprising: A scene analysis module composed of a distributed microphone array, a ToF laser radar and an inertial measurement unit, for real-time collection of environmental acoustic characteristics and user behavior data, outputting spatial reflection parameters, background noise spectrum and user real-time position coordinates; A feature extraction module, a deep neural network processor integrated with a multi-modal sensor fusion interface, receives the output data of the scene analysis module, generates a hearing preference parameter and a scene classification label; A sound field optimization module, an FPGA logic unit configured with a sound field reconstruction algorithm core, dynamically optimizes the sound wave propagation path according to the user's head orientation angle and position coordinates, and generates the boundary constraint conditions of the virtual listening sweet spot; An audio effect generation module, including a programmable digital filter bank and a dynamic gain distribution circuit DSP processing core, receives the classification label of the feature extraction module and the boundary parameter of the sound field optimization module, and outputs an optimal audio driving signal with adjusted phase delay and energy distribution; The scene analysis module is connected to the feature extraction module through a PCIe bus, the sound field optimization module receives the intermediate layer feature tensor of the feature extraction module through a high-speed parallel interface, and transmits the reconstruction parameters to the audio effect generation module through an AES / EBU digital audio interface.

8. The adaptive sound management system with a scene analysis module of claim 7, wherein, The scene analysis module includes the following sub-units: An acoustic sensor group composed of a 32-channel ring microphone array extracts a multipath reflection time series through a beamforming algorithm; A laser radar group uses a multi-wavelength ToF sensor to construct a reflection coefficient matrix of spatial obstacles; A data fusion unit generates a three-dimensional sound field propagation model with sub-microsecond synchronization accuracy by aligning the timestamps of the acoustic sensor group data and the point cloud information of the laser radar group; The data of the acoustic sensor group and the laser radar group are input into the data fusion unit through an LVDS interface, and the fusion result is transmitted to the feature extraction module through a gigabit Ethernet protocol.

Images