Adaptive sound effect management method and system with scene analysis module

Abstract

The invention discloses a self-adaptive sound effect management method and system with a scene analysis module, and relates to the field of self-adaptive sound effect management, and the method comprises the following steps: S1, collecting environmental acoustic characteristics and user behavior data in real time through the scene analysis module; s2, dynamically extracting user hearing preference parameters and scene classification labels; s3, constructing a virtual audition sweet zone matched with the physical space; and S4, dynamically adjusting the phase delay and the energy distribution proportion of each frequency band, and generating an optimal loudness equalization curve meeting the multi-scene constraint. According to the method, adaptive matching of sound field characteristics is realized in a complex dynamic scene through a multi-level perception-optimization closed-loop architecture, acoustic feature extraction and physical space modeling are deeply fused, the technical boundary of traditional pure signal processing is broken through, and the problem of multi-target conflict is solved based on cross-modal collaborative learning and a global optimization algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of adaptive sound effect management, and more specifically, relates to an adaptive sound effect management method having a scene analysis module. The present invention also relates to an adaptive sound effect management system having a scene analysis module. Background Art

[0002] When using wireless devices, the sound quality often needs to be adjusted according to the changing environment. The user's environment is often in a state of constant change. When the user moves from a quiet place to a noisy one, in order to obtain better sound quality and improve voice clarity, the user often needs to adjust the volume according to the environment.

[0003] For example, the method and device for adaptively adjusting sound effects disclosed in patent CN102436821A employs an adaptive sound effects management system that uses an environment adaptation strategy based on a fixed acoustic model. However, its core flaws are reflected in the following aspects:

[0004] Environmental perception is limited in dimension, relying on data collection from a single microphone array or limited sensors. It fails to jointly model acoustic features and spatial geometric constraints, causing the sound field reconstruction process to ignore obstacle reflections and dynamic factors such as user movement trajectories.

[0005] The feature extraction mechanism is rigid. The frequency feature extraction method does not incorporate the deep correlation of multimodal behavioral data. In particular, in dynamic scene classification, there is a lack of coordinated analysis of user visual attention and body movements, resulting in a mismatch between scene labels and auditory requirements.

[0006] The sound field optimization strategy is fragmented, using either passive reverberation suppression or beamforming in independent operation modes, without establishing a global optimization framework for sound field synthesis and speaker distribution;

[0007] Based on the above, we propose an adaptive sound management method and system with a scene analysis module to effectively address the above shortcomings. Summary of the Invention

[0008] The purpose of the present invention is to address the shortcomings of the existing technology and propose an adaptive sound management method and system with a scene analysis module. Through a multi-level perception-optimization closed-loop architecture, it realizes adaptive matching of sound field characteristics in complex dynamic scenes, deeply integrates acoustic feature extraction with physical space modeling, breaks through the technical boundaries of traditional pure signal processing, and solves the problem of multi-objective conflict based on cross-modal collaborative learning and global optimization algorithms.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] The adaptive sound effect management method with a scene analysis module includes the following steps:

[0011] S1. Real-time collection of environmental acoustic characteristics and user behavior data through a scene analysis module. The environmental acoustic characteristics include spatial reflection parameters, background noise spectrum, and sound source distribution characteristics.

[0012] S2. Dynamically extract user hearing preference parameters and scene classification labels based on a deep neural network based on multimodal sensor fusion. The hearing preference parameters include frequency response sensitivity, dynamic range threshold, and sound field diffusion preference.

[0013] S3: Based on the user's real-time location coordinates and head orientation angle, the sound field reconstruction algorithm is used to optimize the distribution of sound wave propagation paths and construct a virtual listening sweet spot that matches the physical space.

[0014] S4. Through a programmable digital filter bank and an adaptive gain controller, combined with the classification labels generated in step S2 and the reconstruction parameters in step S3, the phase delay and energy distribution ratio of each frequency band are dynamically adjusted to generate an optimal loudness equalization curve that meets the constraints of multiple scenarios.

[0015] Preferably, the environmental acoustic feature acquisition in step S1 further includes: integrating the orientation data of the inertial measurement unit and the obstacle distance information of the millimeter wave radar through a multimodal sensor fusion algorithm to establish a three-dimensional sound wave propagation obstacle model; specifically including:

[0016] S11. Use particle filtering algorithm to fuse the six degrees of freedom posture data from the IMU to generate a real-time probability distribution map of the user's head movement;

[0017] S12. Construct a reflection coefficient matrix of spatial obstacles using the flight time measurement values ​​of the millimeter-wave radar, and calculate the path attenuation factors of the direct sound and the reflected sound based on the geometric acoustic model;

[0018] S13. Combining the output parameters of steps S11 and S12, applying the Markov decision process to dynamically predict the optimal diffraction path of the sound wave in a complex obstacle environment, so as to reduce multipath interference and increase the proportion of direct sound energy.

[0019] Preferably, the deep neural network in step S2 includes an adversarial training mechanism, specifically:

[0020] The generator network uses a bidirectional LSTM structure to synthesize personalized preference feature vectors based on historical auditory data;

[0021] The discriminator network adopts a residual convolution architecture to jointly discriminate the authenticity of the generated data by the spectrum map and HRTF parameters;

[0022] The design includes a dynamic weight adaptive loss function to reduce the user preference prediction error of the model in limited sample scenarios.

[0023] Preferably, the virtual listening sweet spot construction process in step S3 includes a dynamic compensation mechanism, and the specific steps are:

[0024] S31. Calculate individual difference parameters of ear canal resonance frequency using a bioacoustic model and establish a personalized frequency response equalization curve.

[0025] S32. Combine the user's pupil tracking data with the brainwave alpha band energy value to evaluate the auditory attention concentration index in real time;

[0026] S33: When attention drift is detected exceeding the threshold, the comb filter effect in the 10-14kHz frequency band is automatically enhanced to improve the perception of sound and image positioning.

[0027] Preferably, the generation of the optimal loudness equalization curve in step S4 also includes a multi-device collaborative optimization process, specifically:

[0028] S41, synchronizing the phase difference of the distributed loudspeaker through the delay compensation protocol of the Bluetooth Mesh network;

[0029] S42. Establish a power allocation strategy for each speaker unit based on the Nash equilibrium theory of game theory to ensure that the sound pressure level uniformity is within ±1.5dB;

[0030] S43. An acoustic fingerprint matching algorithm is used to eliminate tonal distortion caused by frequency response differences between devices, and collaborative optimization improves the sound field coherence of the multi-channel system.

[0031] Preferably, the deep neural network in step S2 adopts a cross-modal attention mechanism, specifically comprising:

[0032] The audio branch network extracts frequency domain feature maps through 3 layers of dilated convolution;

[0033] The visual branch network extracts scene semantic embedding vectors through a two-stream TSN model;

[0034] The cross-modal collaboration module implements spectral domain attention weight allocation, and its attention coefficient calculation formula is:

[0035] in, is the u-th audio feature unit, is the vth visual feature unit, W a is the trainable projection matrix, α uv is the attention coefficient, σ is the activation function; the calculation mechanism of the above formula improves the classification accuracy of complex scenes.

[0036] Preferably, the sound field reconstruction algorithm in step S3 includes a collaborative optimization process of beamforming and sound field synthesis, specifically including:

[0037] S31, generating a personalized head-related transfer function according to the user's auricle HRTF model;

[0038] S32. Calculate the optimal beam weight vector using the minimum variance distortion-free response algorithm:

[0039] Among them, ω is the optimal beam weight vector, which realizes the directional control of the sound beam in space by weighting the complex number of the output signal of each speaker unit; R is the noise covariance matrix, which is calculated by R -1 The calculation realizes the inverse compensation of the environmental noise characteristics and improves the signal-to-noise ratio. a(θ) is the desired direction array manifold vector, which describes the spatial transfer characteristics of the loudspeaker array in the θ direction. After coupling with the HRTF parameters, it generates the array-human ear joint transfer function.

[0040] S33, combining HRTF parameters and user real-time coordinates, iteratively solving the boundary conditions of the Helmholtz equation in the near-field sound field model, and a hybrid iterative algorithm used in the near-field sound field model;

[0041] By deeply coupling the real-time inverse operation of the noise covariance matrix with the HRTF parameters, adaptive beamforming of ambient noise and precise positioning of near-field sound images are achieved in the three-dimensional audio field.

[0042] An adaptive sound effects management system with a scene analysis module, which is used to implement the above method, includes:

[0043] The scene analysis module, consisting of a distributed microphone array, a ToF lidar, and an inertial measurement unit, is used to collect environmental acoustic characteristics and user behavior data in real time, and output spatial reflection parameters, background noise spectrum, and real-time user location coordinates;

[0044] A feature extraction module, a deep neural network processor integrated with a multimodal sensor fusion interface, receives the output data of the scene analysis module and generates auditory preference parameters and scene classification labels;

[0045] The sound field optimization module, equipped with an FPGA logic unit that contains the core of the sound field reconstruction algorithm, dynamically optimizes the sound wave propagation path based on the user's head orientation angle and position coordinates, and generates the boundary constraints of the virtual listening sweet spot;

[0046] The sound effect generation module includes a DSP processing core with a programmable digital filter bank and a dynamic gain distribution circuit, receives the classification labels from the feature extraction module and the boundary parameters from the sound field optimization module, and outputs an optimal audio driving signal that adjusts phase delay and energy distribution;

[0047] Among them, the scene analysis module is connected to the feature extraction module through the PCIe bus, the sound field optimization module receives the intermediate layer feature tensor of the feature extraction module through the high-speed parallel interface, and passes the reconstruction parameters to the sound effect generation module through the AES / EBU digital audio interface.

[0048] Preferably, the scene analysis module includes the following subunits:

[0049] The acoustic sensor group consists of a 32-channel circular microphone array that extracts multipath reflection time series using a beamforming algorithm;

[0050] The LiDAR group uses a multi-wavelength ToF sensor to construct a reflection coefficient matrix of spatial obstacles;

[0051] Data fusion unit: This unit aligns the timestamp data of the acoustic sensor group with the point cloud information of the lidar group to generate a three-dimensional sound field propagation model with sub-microsecond synchronization accuracy.

[0052] The data from the acoustic sensor group and the lidar group are input into the data fusion unit through the LVDS interface, and the fusion results are transmitted to the feature extraction module via the Gigabit Ethernet protocol.

[0053] Technical effects and advantages of the present invention: Compared with the prior art, the adaptive sound effect management method and system with a scene analysis module provided by the present invention has the following advantages:

[0054] Deep integration of multi-dimensional environmental perception, through the multimodal data fusion of distributed microphone arrays, ToF lidar, and inertial measurement units, enables geometric constraint modeling of sound wave propagation paths and physical modeling of dynamic obstacle reflection characteristics, fundamentally addressing the dimensional limitations of environmental parameter collection in traditional methods.

[0055] Cross-modal feature collaborative learning leverages the adversarial training mechanism of deep neural networks and a cross-modal attention model to simultaneously explore the deep correlation between user listening preferences and behavioral characteristics. This breaks through the cognitive boundaries of traditional solutions that analyze audio features independently, significantly improving the semantic consistency of scene classification.

[0056] Globally optimize sound field synthesis. Based on hybrid optimization theories, such as the joint solution of Nash equilibrium and boundary element method, this method constructs an integrated model of sound field reconstruction and equipment collaboration. This method achieves optimal energy allocation under multi-objective constraints while ensuring the accuracy of 3D sound and image positioning, overcoming the inherent defects of local optimization in traditional methods.

[0057] The low-latency, high-energy-efficiency architecture design ensures the system's real-time responsiveness in complex scenarios through triple optimization of a layered clock synchronization mechanism, a high-speed bus protocol between modules, and a dynamic power management strategy. It also breaks the energy efficiency bottleneck of traditional architectures, achieving compatibility between professional-grade sound quality and the battery life of mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of the adaptive sound effect management method with a scene analysis module of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0060] The present invention provides an adaptive sound effects management method and system with a scene analysis module. Through a multi-level perception-optimization closed-loop architecture, it achieves adaptive matching of sound field characteristics in complex dynamic scenes:

[0061] The deep integration of acoustic feature extraction and physical space modeling breaks through the technical boundaries of traditional pure signal processing; based on cross-modal collaborative learning and global optimization algorithms, it solves the problem of multi-objective conflicts; through hardware-algorithm collaborative design, it ensures the system's comprehensive advantages in real-time and power consumption control; this solution provides the field of intelligent audio with a universal sound field management paradigm for dynamic scenarios, which has both theoretical universality and engineering application value.

[0062] like Figure 1 As shown, the adaptive sound effect management method with a scene analysis module includes the following steps:

[0063] S1. Real-time collection of environmental acoustic features and user behavior data through a scene analysis module, wherein the environmental acoustic features include spatial reflection parameters, background noise spectrum, and sound source distribution characteristics; the environmental acoustic feature collection in step S1 further includes: integrating the orientation data of an inertial measurement unit (IMU) and the obstacle distance information of a millimeter-wave radar through a multimodal sensor fusion algorithm to establish a three-dimensional sound wave propagation obstacle model; specifically, including:

[0064] a) Use a particle filter algorithm to fuse the six degrees of freedom pose data from the IMU to generate a real-time probability distribution map of the user's head movement;

[0065] b) Constructing a reflection coefficient matrix for spatial obstacles using the time-of-flight measurements of the millimeter-wave radar and calculating the path attenuation factors for direct and reflected sound based on a geometric acoustic model;

[0066] c) combining the output parameters of steps a) and b) and applying a Markov decision process to dynamically predict the optimal diffraction path of the sound wave in a complex obstacle environment;

[0067] In step S1, the scene analysis module uses a distributed microphone array to work in conjunction with the ToF lidar to calculate the spatial reflection parameters through the space-time correlation of multi-channel signals, specifically including: establishing a geometric constraint equation for the sound wave propagation path:

[0068] Where cij represents the coupling coefficient between the i-th microphone and the j-th reflecting surface, m i is the spatial coordinate of the microphone, s k is the three-dimensional coordinate of the kth sound source, λ k is the wavelength of the sound wave at the corresponding frequency, θ ik is the incident angle of the sound source, τ jk is the time delay of the sound wave reaching the microphone through the jth reflection surface. The calculation method reduces the error in modeling the spatial reflection characteristics.

[0069] S2. A deep neural network based on multimodal sensor fusion dynamically extracts user hearing preference parameters and scene classification labels. The hearing preference parameters include frequency response sensitivity, dynamic range threshold, and sound field diffusion preference. The deep neural network in step S2 includes an adversarial training mechanism. The specific implementation method is as follows:

[0070] The generator network uses a bidirectional LSTM structure to synthesize personalized preference feature vectors based on historical auditory data;

[0071] The discriminator network adopts a residual convolution architecture to jointly discriminate the authenticity of the generated data by the spectrum map and HRTF parameters;

[0072] The design includes a dynamic weight adaptive loss function:

[0073]

[0074] in, represents the expected discrimination value of the real sample, x is the sample in the real auditory dataset (including spectrum and HRTF parameters), p data This is the real data distribution; it is used to drive the discriminator to improve its ability to recognize real user preference data and ensure that the feature vector output by the generator conforms to the real physiological auditory characteristics;

[0075] The latent variable input follows a Gaussian distribution pz, and G(z) is the preference feature vector synthesized by the generator. By minimizing the discriminant score of the generated data, the generator is prompted to learn the statistical characteristics of the user's preference latent space, breaking through the traditional supervised learning's reliance on labeled data.

[0076] is a random point sampled on the line connecting the real data and the generated data, For the discriminator The gradient at , forces the Lipschitz constant of the discriminator to be close to 1, avoiding training collapse and solving the collapse problem of the traditional GAN ​​model.

[0077] Through the above, modal compatibility is enhanced: the introduction of the gradient penalty term enables the discriminator to maintain stable training dynamics in the high-dimensional mixed data space of spectral time-frequency characteristics and HRTF spatial parameters, avoiding feature confusion caused by data distribution shift;

[0078] Breakthrough in small sample generalization capabilities: Through a latent variable generation mechanism and an adversarial game framework, this approach effectively mines the deep correlations of limited user behavior data, generating pseudo samples that conform to personalized listening distributions, significantly improving prediction accuracy in cold start scenarios.

[0079] Physically constrained embedding: The discriminator performs a joint discriminative operation on HRTF parameters, hard-coding the physiological characteristics of the human auditory organ into the adversarial training process to ensure that the generated feature vectors conform to the physical laws of sound wave propagation.

[0080] Optimized training efficiency: The dynamic weight coefficient λ is adaptively adjusted according to the discriminator gradient amplitude, which speeds up network convergence without manual parameter adjustment.

[0081] It should be noted that the deep neural network in step S2 adopts a cross-modal attention mechanism, which specifically includes:

[0082] The audio branch network extracts frequency domain feature maps through 3 layers of dilated convolution;

[0083] The visual branch network extracts scene semantic embedding vectors through a two-stream TSN model;

[0084] The cross-modal collaboration module implements spectral domain attention weight allocation, and its attention coefficient calculation formula is:

[0085] in, is the u-th audio feature unit, is the vth visual feature unit, W a is the trainable projection matrix, α uv is the attention coefficient, σ is the activation function; the calculation mechanism of the above formula improves the classification accuracy of complex scenes.

[0086] S3. Based on the user's real-time position coordinates and head orientation angle, a sound field reconstruction algorithm is called to optimize the sound wave propagation path distribution and construct a virtual listening sweet spot that matches the physical space. The virtual listening sweet spot construction process in step S3 includes a dynamic compensation mechanism, and the specific implementation steps are as follows:

[0087] 1) Calculate individual differences in ear canal resonance frequency using a bioacoustic model and establish a personalized frequency response equalization curve;

[0088] 2) Combine the user's pupil tracking data with the energy value of the alpha band of the brain wave to evaluate the auditory attention concentration index in real time;

[0089] 3) When attention drift is detected exceeding a threshold, the comb filtering effect in the 10-14kHz frequency band is automatically enhanced to improve the perception of sound and image positioning; the compensation mechanism extends the duration of the immersive audio experience by 2.3 times.

[0090] Furthermore, the sound field reconstruction algorithm in step S3 includes a collaborative optimization process of beamforming and sound field synthesis, specifically including:

[0091] S31, generating a personalized head-related transfer function according to the user's auricle HRTF model;

[0092] S32. Calculate the optimal beam weight vector using the minimum variance distortion-free response algorithm:

[0093] Among them, ω is the optimal beam weight vector, which realizes the directional control of the sound beam in space by weighting the complex number of the output signal of each speaker unit; R is the noise covariance matrix, which is calculated by R -1 The calculation realizes the inverse compensation of the environmental noise characteristics and improves the signal-to-noise ratio. a(θ) is the desired direction array manifold vector, which describes the spatial transfer characteristics of the loudspeaker array in the θ direction. After coupling with the HRTF parameters, it generates the array-human ear joint transfer function.

[0094] S33, combining HRTF parameters and user real-time coordinates, iteratively solving the boundary conditions of the Helmholtz equation in the near-field sound field model, and a hybrid iterative algorithm used in the near-field sound field model;

[0095] By deeply coupling the real-time inverse operation of the noise covariance matrix with the HRTF parameters, adaptive beamforming of ambient noise and precise positioning of near-field sound images are achieved in the three-dimensional audio field.

[0096] In addition, the adaptive waveguide model is used to optimize the sound field propagation path, and its core calculation equation is:

[0097] Among them, L eff is the effective sound field propagation path length, Ai represents the amplitude attenuation factor of the i-th reflection path, k is the wave number, r i is the Euclidean distance from the user position to the i-th speaker, is the phase compensation term, which enhances the direct sound through constructive interference in the waveguide path, thereby improving speech clarity in a reverberant environment and reducing the comb filtering effect to an imperceptible level.

[0098] S4. By combining the classification labels generated in step S2 and the reconstruction parameters in step S3, the programmable digital filter bank and the adaptive gain controller dynamically adjust the phase delay and energy distribution ratio of each frequency band to generate an optimal loudness equalization curve that meets the constraints of multiple scenarios. The generation of the optimal loudness equalization curve in step S4 also includes a multi-device collaborative optimization process, which is specifically implemented as follows:

[0099] a) Synchronize the phase difference of distributed loudspeakers through the delay compensation protocol of Bluetooth Mesh network;

[0100] b) Establish a power allocation strategy for each speaker unit based on the Nash equilibrium theory of game theory to ensure that the sound pressure level uniformity is within ±1.5dB;

[0101] c) Use acoustic fingerprint matching algorithm to eliminate the tonal distortion caused by frequency response differences between devices.

[0102] In addition, the sound field reconstruction algorithm includes a multi-user collaborative processing mechanism, and the specific implementation method is as follows:

[0103] 1) Real-time tracking of the coordinates and movement speeds of multiple users through an ultra-wideband positioning system;

[0104] 2) Apply the Voronoi diagram segmentation algorithm to divide the priority radiation area of ​​each user;

[0105] 3) Design a time-domain staggered beam focusing strategy to distribute the sound energy within the 30-100ms time window to the auditory perception area of ​​high-priority users; this mechanism improves individual speech recognition in multi-user concurrent scenarios.

[0106] The digital filter bank in step S4 adopts a dynamic coefficient update strategy, and its parameter adjustment rule is:

[0107] Among them, ΔG m is the gain compensation value required for the mth critical frequency band, α is the frequency response matching weight coefficient, H opt (f) is the optimized target transfer function that complies with international electroacoustic standards, H env (f) is the environmental transfer function measured by the scene analysis module, f Hm / f Lmis the lower / upper cutoff frequency of the mth frequency band, β is the noise robustness adjustment factor, and its value is positively correlated with the room reverberation time. m is the signal-to-noise ratio of the mth frequency band;

[0108] The above formula solves the following problems simultaneously through a weighted combination of frequency response matching and noise suppression:

[0109] Frequency domain distortion correction: Generate accurate compensation curves for environmental acoustic defects;

[0110] Time domain noise suppression: suppresses reverberation tails and background noise in low signal-to-noise ratio areas;

[0111] Perceptual consistency preservation: ΔG based on equal loudness curve constraints m Allocation strategy to ensure that the compensation results are consistent with the auditory characteristics of the human ear.

[0112] This embodiment further proposes an adaptive sound effect management system having a scene analysis module, which is used to implement the above method and includes:

[0113] Scene Analysis Module: This module consists of a distributed microphone array, a ToF lidar, and an inertial measurement unit. It collects environmental acoustic characteristics and user behavior data in real time, and outputs spatial reflection parameters, background noise spectrum, and the user's real-time location coordinates. The scene analysis module includes the following subunits:

[0114] Acoustic sensor group: Consists of a 32-channel circular microphone array with a sampling rate of 192kHz, and uses a beamforming algorithm to extract multipath reflection time series;

[0115] LiDAR group: uses a multi-wavelength ToF sensor with a detection accuracy of ±5mm to construct a reflection coefficient matrix of spatial obstacles;

[0116] Data fusion unit: This unit fuses the timestamp-aligned data from the acoustic sensor group with the point cloud information from the lidar group through an extended Kalman filter to generate a three-dimensional sound field propagation model with sub-microsecond synchronization accuracy. The model has an error rate of less than 2.3%. The data from the acoustic sensor group and the lidar group are input into the data fusion unit through the LVDS interface, and the fusion result is transmitted to the feature extraction module via the Gigabit Ethernet protocol.

[0117] Feature extraction module: A deep neural network processor with an integrated multimodal sensor fusion interface receives the output data of the scene analysis module and generates auditory preference parameters and scene classification labels. The feature extraction module includes the following processing links:

[0118] Audio feature extraction link: A third-order anti-aliasing FIR filter bank is used to divide the input acoustic signal into 1 / 3 octave bands, and Mel-frequency cepstral coefficients and dynamic range compression parameters are extracted through time-frequency transformation.

[0119] Behavior perception link: Deploy a two-branch graph convolutional network to process the six-axis motion data from the IMU and the visual attention heat map from the pupil tracking sensor;

[0120] Cross-modal fusion unit: Apply the gated attention mechanism to weightedly aggregate audio and behavioral features. The attention weight calculation formula is:

[0121] in, is the i-th audio feature vector, is the jth behavior feature vector, W g is the gated weight matrix; the processing link achieves zero-latency interaction of multimodal features through a shared memory architecture, thereby reducing the overall inference latency.

[0122] Sound Field Optimization Module: This module, equipped with an FPGA logic unit that includes the core of the sound field reconstruction algorithm, dynamically optimizes the sound wave propagation path based on the user's head orientation angle and position coordinates, generating boundary constraints for the virtual listening sweet spot. The architecture of this sound field optimization module includes:

[0123] Personalized HRTF database: stores more than 500 sets of head-related transfer function measurement data, each data set contains impulse responses of 360° spatial orientation at a 48kHz sampling rate;

[0124] Real-time sound field solver: Based on the CUDA-accelerated boundary element method engine, it dynamically calculates the sound pressure distribution at the user's ear position. The calculation expression is:

[0125] Where Q n is the equivalent intensity of the nth virtual source, r n is the virtual source coordinate, k is the acoustic wave number, is the scaling factor of the three-dimensional free-field Green's function, which is used to correct the natural attenuation caused by the diffusion of spherical waves. Its inverse relationship ensures the dimensional consistency of the far-field and near-field models.

[0126] Adaptive beam controller: uses a hybrid MUSIC algorithm to calculate the optimal loudspeaker excitation weight parameters, and writes the calibration parameters into the register of the power amplifier module through the I²C bus. The control cycle is accurate to 1ms level.

[0127] The above formula combines the personalized HRTF database with the virtual source reconstruction technology to achieve two major breakthroughs in the sound field optimization module: Accurate sound image modeling: Through the virtual source equivalent intensity Q n With spatial coordinate r n Dynamic matching accurately simulates the propagation path of sound waves after reflection and diffraction by complex obstacles, making the three-dimensional sound pressure distribution in the virtual listening sweet spot highly consistent with the characteristics of the real physical space;

[0128] Real-time computing performance: complex phase terms The GPU parallel computing architecture, combined with the CUDA-accelerated boundary element method engine, meets the real-time requirements of a 1ms control cycle, breaking through the computing bottleneck of traditional acoustic simulation.

[0129] Multi-physics field coupling: The frequency-varying characteristics of wave number k and the dynamic adjustment of digital filter bank form cross-domain coupling to achieve quantitative improvement of speech clarity index. Ensure numerical stability when switching between far / near field models.

[0130] Adaptive beam steering: This module uses a hybrid MUSIC algorithm to calculate the optimal speaker excitation weight parameters and writes the calibration parameters to the amplifier module's registers via the I²C bus. The control cycle is accurate to 1ms. This module achieves high-performance beam steering through the following technical improvements:

[0131] Multiresolution spectral estimation architecture:

[0132] Based on the forward / backward spatial smoothing processing of the hybrid MUSIC algorithm, the azimuth ambiguity problem of broadband sound sources is resolved;

[0133] Combined with covariance matrix reconstruction technology to eliminate the mutual coupling effect between channels of distributed loudspeaker array;

[0134] Dynamically adjust the spatial spectrum peak search range to adapt to the position changes of mobile users in real time;

[0135] Calibration parameter writing mechanism:

[0136] The power amplifier module is configured with a dedicated register group, including a frequency division coefficient register, a gain adjustment register, and a delay compensation register;

[0137] Microsecond-level parameter synchronization is achieved through the multi-master arbitration protocol of the I²C bus, and redundancy checks are implemented to prevent signal distortion;

[0138] Supports burst transmission mode, and can complete batch update of parameters of 128 speaker nodes in a single communication cycle;

[0139] Precision timing management:

[0140] Adopting a hierarchical clock tree architecture, the master FPGA outputs nanosecond-level accurate trigger pulse sequences;

[0141] The integrated dynamic phase-locked loop module eliminates the cumulative phase error between multiple clock domains and ensures the stability of the 1ms control cycle.

[0142] The sound effect generation module, a DSP processing core containing a programmable digital filter bank and a dynamic gain distribution circuit, receives the classification labels from the feature extraction module and the boundary parameters from the sound field optimization module, and outputs the optimal audio drive signal with adjusted phase delay and energy distribution. This module achieves multi-objective optimization through the following core technologies:

[0143] Filter bank dynamic configuration mechanism:

[0144] The polyphase filter bank design supports 1 / 24 octave frequency band subdivision and can dynamically load 128 sets of time-varying impulse response parameters. The filter coefficients of each subband are reconstructed in real time according to the boundary parameters of the sound field optimization module, prioritizing compensation for the user's auditory sensitivity bands. The stopband attenuation depth is set in conjunction with the auditory masking effect model to avoid auditory fatigue caused by high-frequency harmonic components.

[0145] Gain allocation strategy:

[0146] Based on a dual-threshold dynamic compression strategy, the system distinguishes energy allocation rules between areas dominated by ambient noise and areas active by speech signals. It introduces an asymmetric clipping function to prevent transient acoustic signal overload while preserving the dynamic fluctuation characteristics of the music signal. It also establishes an energy prediction model based on historical scene labels, pre-allocating gain margins in each frequency band to reduce adjustment delays.

[0147] Cross-domain collaborative optimization:

[0148] The system receives sound field reconstruction parameters via an AES / EBU interface and performs time-domain pre-compensation and frequency-domain interpolation of phase delay. A multi-layer pipeline architecture is used to split the signal processing chain into three parallel stages: frame synchronization, frequency band slicing, and dynamic equalization. An on-chip error correction unit is integrated, and BCH encoding is used to ensure the transmission integrity of the distributed loudspeaker drive signal.

[0149] Among them, the scene analysis module is connected to the feature extraction module through the PCIe bus, the sound field optimization module receives the intermediate layer feature tensor of the feature extraction module through the high-speed parallel interface, and passes the reconstruction parameters to the sound effect generation module through the AES / EBU digital audio interface.

[0150] The synergistic effects of the above system modules are as follows:

[0151] Real-time performance guarantee: The parameter update rate of the beam controller and the frame processing cycle of the sound effect module are precisely aligned through timestamps;

[0152] Resource reuse mechanism: The two share the scene classification labels of the feature extraction module and reuse the azimuth index data of the HRTF database;

[0153] Multi-objective optimization: The spatial acoustic energy focusing of beamforming and the frequency domain energy distribution of sound effect generation form a joint cost function to solve the closed loop;

[0154] Hardware architecture adaptation: High-parallelism FPGA and ultra-low latency DSP chips are used to match the characteristics of their respective computation-intensive and control-intensive tasks.

[0155] In summary, the present invention has the following effects:

[0156] Deep fusion of multi-dimensional environmental perception: Through the multimodal data fusion of distributed microphone arrays, Time of Flight LiDAR, and inertial measurement units, geometric constraint modeling of sound wave propagation paths and physical modeling of dynamic obstacle reflection characteristics are achieved, fundamentally addressing the dimensional limitations of environmental parameter collection in traditional methods.

[0157] Cross-modal feature collaborative learning: Leveraging the adversarial training mechanism of deep neural networks and a cross-modal attention model, this technology simultaneously explores the deep correlation between user listening preferences and behavioral characteristics, breaking through the cognitive limitations of traditional approaches that rely on independent analysis of audio features and significantly improving the semantic consistency of scene classification.

[0158] Globally optimized sound field synthesis: Based on hybrid optimization theories, such as the joint solution of Nash equilibrium and boundary element method, an integrated model of sound field reconstruction and equipment collaboration is constructed. This method achieves optimal energy allocation under multi-objective constraints while ensuring the accuracy of 3D sound image positioning, overcoming the inherent defects of local optimization in traditional methods.

[0159] Low-latency, high-energy-efficiency architecture design: Through triple optimization of a layered clock synchronization mechanism, a high-speed bus protocol between modules, and a dynamic power management strategy, the system ensures real-time responsiveness in complex scenarios while breaking the energy efficiency bottleneck of traditional architectures, achieving compatibility between professional-grade sound quality and mobile device battery life.

[0160] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An adaptive sound effect management method with a scene analysis module, characterized in that: The steps include: S1. Real-time collection of environmental acoustic characteristics and user behavior data through a scene analysis module. The environmental acoustic characteristics include spatial reflection parameters, background noise spectrum, and sound source distribution characteristics. S2. Dynamically extract user hearing preference parameters and scene classification labels based on a deep neural network based on multimodal sensor fusion. The hearing preference parameters include frequency response sensitivity, dynamic range threshold, and sound field diffusion preference. S3: Based on the user's real-time location coordinates and head orientation angle, the sound field reconstruction algorithm is used to optimize the distribution of sound wave propagation paths and construct a virtual listening sweet spot that matches the physical space. S4. Through a programmable digital filter bank and adaptive gain controller, the phase delay and energy distribution ratio of each frequency band are dynamically adjusted to generate an optimal loudness equalization curve that meets the constraints of multiple scenarios.

2. The adaptive sound effect management method with a scene analysis module according to claim 1, characterized in that: The environmental acoustic feature acquisition in step S1 further includes: integrating the orientation data of the inertial measurement unit and the obstacle distance information of the millimeter wave radar through a multimodal sensor fusion algorithm to establish a three-dimensional sound wave propagation obstacle model; specifically, including: S11. Use particle filtering algorithm to fuse the six degrees of freedom posture data from the IMU to generate a real-time probability distribution map of the user's head movement; S12. Construct a reflection coefficient matrix of spatial obstacles using the flight time measurement values ​​of the millimeter-wave radar, and calculate the path attenuation factors of the direct sound and the reflected sound based on the geometric acoustic model; S13. Combining the output parameters of steps S11 and S12, applying the Markov decision process to dynamically predict the optimal diffraction path of the sound wave in a complex obstacle environment, so as to reduce multipath interference and increase the proportion of direct sound energy.

3. The adaptive sound effect management method with a scene analysis module according to claim 1, characterized in that: The deep neural network in step S2 includes an adversarial training mechanism, specifically: The generator network uses a bidirectional LSTM structure to synthesize personalized preference feature vectors based on historical auditory data; The discriminator network adopts a residual convolution architecture to jointly discriminate the authenticity of the generated data by the spectrum map and HRTF parameters; The design includes a dynamic weight adaptive loss function to reduce the user preference prediction error of the model in limited sample scenarios.

4. The adaptive sound effect management method with a scene analysis module according to claim 1, characterized in that: The virtual listening sweet spot construction process in step S3 includes a dynamic compensation mechanism, and the specific steps are as follows: S31. Calculate individual difference parameters of ear canal resonance frequency using a bioacoustic model and establish a personalized frequency response equalization curve. S32. Combine the user's pupil tracking data with the brainwave alpha band energy value to evaluate the auditory attention concentration index in real time; S33. When attention drift is detected exceeding the threshold, the comb filter effect in the 10-14kHz frequency band is automatically enhanced to improve the perception of sound and image positioning.

5. The adaptive sound effect management method with a scene analysis module according to claim 1, characterized in that: The generation of the optimal loudness equalization curve in step S4 also includes a multi-device collaborative optimization process, specifically: S41, synchronizing the phase difference of the distributed loudspeaker through the delay compensation protocol of the Bluetooth Mesh network; S42. Establish a power allocation strategy for each speaker unit based on the Nash equilibrium theory of game theory to ensure that the sound pressure level uniformity is within ±1.5dB; S43. An acoustic fingerprint matching algorithm is used to eliminate tonal distortion caused by frequency response differences between devices, and collaborative optimization improves the sound field coherence of the multi-channel system.

6. The adaptive sound effect management method with a scene analysis module according to claim 3, characterized in that: The deep neural network in step S2 adopts a cross-modal attention mechanism, which specifically includes: The audio branch network extracts frequency domain feature maps through 3 layers of dilated convolution; The visual branch network extracts scene semantic embedding vectors through a two-stream TSN model; The cross-modal collaboration module implements spectral domain attention weight allocation, and its attention coefficient calculation formula is: in, is the u-th audio feature unit, is the vth visual feature unit, W a is the trainable projection matrix, α uv is the attention coefficient, σ is the activation function; the calculation mechanism of the above formula improves the classification accuracy of complex scenes.

7. The adaptive sound effect management method with a scene analysis module according to claim 1, characterized in that: The sound field reconstruction algorithm in step S3 includes a collaborative optimization process of beamforming and sound field synthesis, specifically including: S31, generating a personalized head-related transfer function according to the user's auricle HRTF model; S32. Calculate the optimal beam weight vector using the minimum variance distortion-free response algorithm: Among them, ω is the optimal beam weight vector, which realizes the directional control of the sound beam in space by weighting the complex number of the output signal of each speaker unit; R is the noise covariance matrix, which is calculated by R -1 The calculation realizes the inverse compensation of the environmental noise characteristics and improves the signal-to-noise ratio. a(θ) is the desired direction array manifold vector, which describes the spatial transfer characteristics of the loudspeaker array in the θ direction. After coupling with the HRTF parameters, it generates the array-human ear joint transfer function. S33, combining HRTF parameters and user real-time coordinates, iteratively solving the boundary conditions of the Helmholtz equation in the near-field sound field model, and a hybrid iterative algorithm used in the near-field sound field model; By deeply coupling the real-time inverse operation of the noise covariance matrix with the HRTF parameters, adaptive beamforming of ambient noise and precise positioning of near-field sound images are achieved in the three-dimensional audio field.

8. An adaptive sound management system with a scene analysis module, characterized in that: The system is used to implement the method described in any one of claims 1 to 7, including: The scene analysis module, consisting of a distributed microphone array, a ToF lidar, and an inertial measurement unit, is used to collect environmental acoustic characteristics and user behavior data in real time, and output spatial reflection parameters, background noise spectrum, and real-time user location coordinates; A feature extraction module, a deep neural network processor integrated with a multimodal sensor fusion interface, receives the output data of the scene analysis module and generates auditory preference parameters and scene classification labels; The sound field optimization module, equipped with an FPGA logic unit that contains the core of the sound field reconstruction algorithm, dynamically optimizes the sound wave propagation path based on the user's head orientation angle and position coordinates, and generates the boundary constraints of the virtual listening sweet spot; The sound effect generation module includes a DSP processing core with a programmable digital filter bank and a dynamic gain distribution circuit, receives the classification labels from the feature extraction module and the boundary parameters from the sound field optimization module, and outputs an optimal audio driving signal that adjusts phase delay and energy distribution; Among them, the scene analysis module is connected to the feature extraction module through the PCIe bus, the sound field optimization module receives the intermediate layer feature tensor of the feature extraction module through the high-speed parallel interface, and passes the reconstruction parameters to the sound effect generation module through the AES / EBU digital audio interface.

9. The adaptive sound effect management system with a scene analysis module according to claim 8, characterized in that: The scene analysis module includes the following subunits: The acoustic sensor group consists of a 32-channel circular microphone array that extracts multipath reflection time series using a beamforming algorithm; The LiDAR group uses a multi-wavelength ToF sensor to construct a reflection coefficient matrix of spatial obstacles; Data fusion unit: This unit aligns the timestamp data of the acoustic sensor group with the point cloud information of the lidar group to generate a three-dimensional sound field propagation model with sub-microsecond synchronization accuracy. The data from the acoustic sensor group and the lidar group are input into the data fusion unit through the LVDS interface, and the fusion results are transmitted to the feature extraction module via the Gigabit Ethernet protocol.

Images