AI-driven vehicle-mounted sound field real-time modeling and voice separation method

Abstract

The invention discloses an AI-driven vehicle-mounted sound field real-time modeling and voice separation method, and relates to the technical field of voice signal processing. A main control unit comprising a time sequence synchronizer, a resource scheduler and a health monitor is constructed. 3D sound field modeling is carried out by adopting a lightweight STCN + bidirectional LSTM network, adaptive updating of the model is realized through EWC incremental learning, and a CNN-LSTM noise classification network and targeted suppression algorithms such as ANF / spectral subtraction are developed. The voice separation module adopts an improved Conv-TasNet architecture, 3D spatial constraint and a multi-task loss function are fused, and low delay is realized under INT8 quantization and pipeline processing. The system dynamically optimizes parameters through a real-time regulation and control unit, supports scene self-adaption, finally achieves a separation effect in a mixed noise scene, reduces the delay of the whole system, and effectively improves the definition and stability of vehicle-mounted voice interaction.

Description

Technical Field

[0001] This invention belongs to the field of speech signal processing technology, specifically relating to an AI-driven method for real-time modeling of vehicle sound fields and speech separation. Background Technology

[0002] Vehicle noise encompasses engine noise, wind noise, road noise, and mixed noise. Existing technologies primarily employ suppression algorithms designed for individual noise sources, but their ability to suppress mixed noise is insufficient, leaving residual noise and causing severe interference during speech separation. Traditional modeling methods struggle to adapt in real time, and the vehicle sound field is affected by operating conditions and interior materials, resulting in significant differences in sound pressure levels across 3D space.

[0003] Traditional sound field modeling methods (such as fixed models based on geometric acoustics) cannot be updated in real time and have a large computational load, making it difficult to meet the real-time requirements of in-vehicle systems.

[0004] In-vehicle speech separation needs to achieve high accuracy with low latency, but traditional methods have high model complexity and excessive latency when running on in-vehicle domain controllers. Lightweight methods (such as pruning / quantization) reduce latency, but accuracy drops significantly, especially in mixed noise scenarios.

[0005] Vehicle-mounted perception data includes microphone audio, vibration sensor data, and CAN bus operating parameters. These data have significantly different sampling frequencies, and traditional synchronization methods (such as simple interpolation) are prone to introducing timestamp errors, leading to inaccurate sound field feature extraction and affecting modeling and separation accuracy. Dynamic changes in vehicle operating conditions can easily degrade model performance. Existing technologies lack effective incremental learning and rollback mechanisms, requiring model retraining and failing to adapt to real-time changes in operating conditions. Furthermore, traditional methods do not incorporate scene adaptation (such as rear-seat voice gain and high-speed wind noise suppression), resulting in a poor user experience.

[0006] To address the aforementioned issues, this invention proposes an AI-driven method for real-time modeling of vehicle sound fields and speech separation. Summary of the Invention

[0007] In order to overcome the shortcomings and deficiencies of the existing technology, the first objective of this invention is to provide an AI-driven method for real-time modeling of vehicle sound field and speech separation; the second objective of this invention is to provide an AI-driven system for real-time modeling of vehicle sound field and speech separation.

[0008] The first objective of this invention is achieved through the following technical solution:

[0009] The AI-driven method for real-time modeling of vehicle sound field and speech separation comprises the following steps:

[0010] Step S101: Real-time acquisition and synchronization of multi-source data: The multi-modal sensor array acquires microphone audio data and supplements it to vibration sensor data through linear interpolation; the CAN data interface reads vehicle operating parameters and bus load rate; the timing synchronizer uses the microphone sampling clock as a reference, and aligns the multi-source data through PPS synchronization + dynamic compensation to ensure that the timestamp error does not exceed the set threshold, and outputs synchronized data including audio, interpolated vibration data, operating parameters, and load rate;

[0011] Step S102: Sound field key feature extraction: The sound field feature extraction engine extracts time domain, frequency domain and 3D spatial domain features from the synchronous data, filters 30 key features through the ECA-Net attention mechanism, and outputs them to the AI ​​sound field modeling engine.

[0012] Step S103: Dynamic sound field modeling and updating: The dynamic sound field modeling engine inputs key features, operating parameters and load rate into a lightweight STCN+bidirectional LSTM network to construct a 3D sound field transfer function; it adaptively updates the model parameters through EWC incremental learning, and extends the update interval to stabilize the operating conditions; it collects measured sound pressure levels at 5 test points to verify the accuracy, outputs the sound field model, and fine-tunes the EWC regularization coefficient when the error exceeds the standard;

[0013] Step S104: Noise Classification and Targeted Suppression: The noise classification and suppression engine inputs the noisy speech into the CNN-LSTM classification network, outputs the probability of 5 types of noise and determines the type; selects the suppression algorithm according to the type, uses the corresponding single algorithm for single noise, and uses the joint algorithm for mixed noise, and outputs the denoised speech signal and noise type identifier;

[0014] Step S105: AI speech separation and quality assessment: The AI ​​speech separation module inputs the denoised speech, sound field model and noise type identifier into the lightweight Conv-TasNet. Single noise enables 6 layers of TCB, mixed noise activates all TCB, and 3D spatial constraints and multi-task loss are incorporated to output the target speech; the separation quality assessment unit calculates SDR, SI-SNR and residual rate and outputs the results.

[0015] Step S106: Real-time control and feedback: When the scenario-based indicators are met, the target voice is output to the human-computer interaction module and the output parameters are adaptively adjusted; when they are not met, the real-time control unit optimizes the runtime parameters through reinforcement learning and feeds them back to S102, S104, and S105. The health monitor monitors the status and calls the historical compliant parameters to roll back the model when an early warning is triggered.

[0016] The second objective of this invention is achieved through the following technical solution:

[0017] The AI-driven vehicle sound field real-time modeling and speech separation system is used to realize the AI-driven vehicle sound field real-time modeling and speech separation method. The system includes a main control unit, a sound field perception module, an AI sound field modeling engine, an AI speech separation module, a real-time control unit, and a human-computer interaction module.

[0018] Main control unit: As the central scheduler of the system, it is deployed on the vehicle domain controller. It has low-latency computing and multi-module collaboration capabilities. It is responsible for the full-process management of sound field perception, AI modeling, voice separation and real-time control, and includes three sub-components: timing synchronizer, resource scheduler and health monitor.

[0019] Sound field perception module: responsible for collecting 3D sound field data and environmental data in the vehicle environment, providing input for AI modeling. It includes a multimodal sensor array consisting of a microphone array and a vibration sensor to collect audio and vibration information; a data preprocessing unit to clean the raw data; and a CAN data interface to obtain dynamic environmental parameters.

[0020] AI sound field modeling engine: Real-time dynamic modeling of vehicle sound field is achieved through lightweight multi-dimensional AI algorithms, which is used for sound field feature extraction, dynamic sound field modeling, and noise classification and suppression.

[0021] AI speech separation module: Based on 3D sound field model and noise suppression results, it achieves accurate separation of target speech and interference noise. It uses spatial constraint separation unit, end-to-end separation network and separation quality evaluation unit to improve separation accuracy and reduce latency.

[0022] Real-time control unit: dynamically adjusts system runtime parameters based on speech separation performance to ensure optimal performance, including an adaptive parameter adjuster to optimize microphone array gain and AI model runtime parameters; feedback learning unit: continuously optimizes AI model using reinforcement learning methods;

[0023] Human-computer interaction module: outputs voice, supports voice command recognition and call noise reduction, and provides a user feedback interface; automatically adjusts parameters according to the scenario.

[0024] Preferably, the main control unit is deployed on the vehicle domain controller, possessing low-latency computing and multi-module collaboration capabilities, and internally includes:

[0025] Timing synchronizer: Through hardware PPS pulse synchronization signal + software dynamic compensation algorithm, it synchronizes the microphone array, vibration sensor and vehicle CAN bus data. In view of the difference in sampling / update frequency of multi-source data, it adopts time axis alignment + data interpolation matching strategy to linearly interpolate the 1kHz raw data of vibration sensor to 48kHz timing. The CAN data compensates for the transmission delay by linear prediction of the parameters of the first 3 frames, ensuring the consistency of timestamps of multi-source data and controlling the synchronization error within the set threshold.

[0026] Resource scheduler: Dynamically allocates CPU / NPU computing power based on vehicle operating conditions and CAN bus load rate. By dynamically adjusting the allocation of CPU / NPU computing power, it ensures that the overall system latency does not exceed the set time. In congested traffic conditions, it automatically reduces the computing power ratio of non-core modules and reserves a set proportion of redundant computing power for synchronous compensation and lightweight model inference.

[0027] Health Monitor: Monitors sensor status and AI model output confidence in real time, triggering fault warnings or model rollback; during rollback, it directly calls the most recent compliant operating condition parameters saved by EWC incremental learning, and the rollback delay does not exceed the set time.

[0028] Preferably, the sound field perception module is responsible for collecting in-vehicle full-domain 3D sound field and environmental data, including:

[0029] Multimodal sensor array: Composed of a 4-8 channel ring + vertical microphone array and a 4-channel vibration sensor;

[0030] Data preprocessing unit: performs DC removal, pre-emphasis and framing processing on the raw acquired data to filter out sensor hardware noise; for the interpolated vibration data, an additional 5ms sliding window smoothing filter is added to avoid high-frequency interference introduced by interpolation;

[0031] CAN data interface: Reads dynamic operating condition data and bus load rate of the vehicle CAN bus as dynamic environmental parameters for sound field modeling; when the bus load is greater than or equal to a set ratio, a data caching mechanism is triggered to avoid data loss.

[0032] Preferably, the AI ​​sound field modeling engine runs on the AI ​​computing core of the main control unit, including:

[0033] Sound field feature extraction engine: Extracts time-domain, frequency-domain, and 3D spatial domain features of the sound field, and combines an attention mechanism to filter key features; optimizes feature dimensions and reduces redundant computation;

[0034] Dynamic sound field modeling engine: It uses lightweight STCN + bidirectional LSTM to build a sound field transfer function model, integrates vehicle operating parameters and CAN bus load rate, and updates the 3D sound field spatial distribution in real time; both STCN convolution kernel and LSTM structure are designed to be lightweight to reduce computing power requirements;

[0035] Noise Classification and Suppression Engine: Based on the CNN-LSTM two-stage classification network, noise types are identified, and adaptive suppression algorithms are designed for single and mixed noise respectively.

[0036] Preferably, the AI ​​speech separation module achieves accurate separation of the target speech based on a 3D sound field model and noise suppression results, including:

[0037] Spatial Constraint Separation Unit: Utilizes the 3D spatial distribution information output by the sound field modeling engine to construct a 3D spatial mask for the target speech, initially filtering out noise from non-target directions; in rear-seat speech scenarios, it automatically increases the vertical mask to the set weight.

[0038] End-to-end separation network: An improved lightweight Conv-TasNet is adopted, which incorporates 3D sound field features and noise classification results to optimize separation weights; for mixed noise scenarios, a lightweight enhancement branch is automatically activated to supplement separation capabilities;

[0039] Separation quality assessment unit: calculates the signal-to-noise ratio, source distortion ratio, and residual mixed noise rate of the separated speech in real time, as feedback indicators for model optimization; and sets accuracy thresholds in a differentiated manner.

[0040] Preferably, in the improved lightweight Conv-TasNet of the AI ​​speech separation module, the encoder is a 1-layer 1D convolution, the separation network is a 6-layer basic temporal convolutional block, a 2-layer lightweight TCB is automatically activated in mixed noise scenarios, and the decoder is a 1-layer transposed convolution; 3D spatial constraints and multi-task loss are introduced; the vehicle real-time requirements are met through model compression, pipeline processing, and dynamic computing power allocation, and the separation quality evaluation threshold is set differently.

[0041] Preferably, the real-time control unit dynamically adjusts the system's runtime parameters based on the speech separation effect, including:

[0042] Adaptive parameter adjuster: Optimizes microphone array gain and AI model runtime parameters based on separation quality assessment results;

[0043] Feedback learning unit: Reinforcement learning is used to continuously iterate and optimize the AI ​​model; the CAN bus load rate weight is introduced into the reward function.

[0044] Preferably, the human-computer interaction module outputs clear voice after separation, supports voice command recognition and call noise reduction, and provides a user feedback interface; in high-speed wind noise scenarios, it automatically increases the set frequency band suppression depth for call noise reduction; in rear-seat voice scenarios, it optimizes the vertical voice output gain.

[0045] Preferably, the noise classification and suppression engine constructs an in-vehicle dataset covering 3 types of single noise and 2 types of typical mixed noise, with a sample ratio of 70% for single noise and 30% for mixed noise. A two-stage classification network of CNN-LSTM is adopted. The CNN feature extraction uses 3 convolutional layers + 1 depthwise separable convolutional layer to extract local features of the noise spectrum. The LSTM temporal modeling uses 2 layers of LSTM to capture the temporal variation of noise. The classification output is processed by a fully connected layer + Softmax to output the probability of 5 types of noise. The type is determined according to the judgment rule. The loss function of the classification network adopts cross-entropy loss. The classification accuracy after training is required to be greater than or equal to a set proportion. And the suppression algorithm is selected according to the classification results.

[0046] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0047] 1. This invention utilizes multi-source data synchronization technology to accurately synchronize multi-source data with errors controlled within a small range; through 3D sound field dynamic modeling, it integrates vehicle operating parameters and CAN bus load rate to update the 3D sound field spatial distribution in real time, accurately depicting the sound propagation law; with the help of a noise classification and suppression engine, it accurately identifies various noise types based on a CNN-LSTM network and adopts targeted suppression algorithms to effectively solve the problem of mixed noise residue, ultimately achieving accurate separation of speech and noise in vehicle scenarios, solving the problems of dynamic noise interference and changing sound fields.

[0048] 2. This invention employs a lightweight algorithm to optimize the network structure, reducing the number of convolutional kernels and model complexity, thereby lowering computational requirements. Combined with a real-time control unit, the system operating parameters are dynamically adjusted based on the speech separation effect. A multi-objective reward function is constructed through reinforcement learning to optimize the AI ​​model. The learning rate is flexibly adjusted according to the CAN bus load rate. While ensuring separation accuracy, the overall system latency is reduced, balancing separation accuracy and real-time performance, and adapting to mixed noise scenarios.

[0049] 3. This invention achieves scene adaptation through the human-computer interaction module. In high-speed wind noise scenarios, it automatically improves the suppression depth of specific frequency bands for call noise reduction. In rear-seat voice scenarios, it optimizes the vertical voice output gain to improve voice clarity. At the same time, the real-time control and model rollback mechanism ensures that the system can operate stably under various working conditions. The health monitor monitors in real time and quickly rolls back the model when abnormalities occur, ensuring the stability of in-vehicle voice interaction. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart of the AI-driven real-time modeling and speech separation method for vehicle sound field of the present invention is shown;

[0052] Figure 2 A block diagram of the system of the present invention is shown;

[0053] Figure 3 The flowchart of the sound field feature extraction engine of the present invention is shown. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more exemplary embodiments. Numerous specific details are provided in the following description to give a full understanding of exemplary embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, steps, etc., can be employed. In other instances, well-known structures, methods, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0056] Example 1:

[0057] Reference Figure 1 As shown, this invention provides an AI-driven method for real-time modeling of in-vehicle sound fields and speech separation, aiming to solve the problems of dynamic noise interference (engine noise, wind noise, road noise, and mixed noise), variable sound field spatial distribution, and the difficulty in balancing speech separation accuracy and real-time performance in in-vehicle scenarios. This method is implemented using the following system, see reference [link / document / reference / etc.]. Figure 2 As shown, the system includes:

[0058] 1.1 Main Control Unit

[0059] As the core scheduling hub of the system, deployed in the vehicle domain controller (such as a cockpit domain controller equipped with NVIDIA Orin), it possesses low-latency computing and multi-module collaboration capabilities, and is responsible for coordinating the entire process of sound field perception, AI modeling, voice separation, and real-time control. Internally, it includes:

[0060] Timing synchronizer: This synchronizer uses a hardware PPS pulse synchronization signal and a software dynamic compensation algorithm to synchronize data from the microphone array, vibration sensor, and vehicle CAN bus (such as vehicle speed and engine speed). To address differences in sampling / update frequencies among the multi-source data, a time axis alignment and data interpolation matching strategy is employed: the 1kHz raw data from the vibration sensor is linearly interpolated to a 48kHz timing sequence (matching the microphone sampling rate), and the CAN data transmission delay is compensated for using linear prediction of the first three frames of parameters. Ultimately, this ensures consistency in timestamps across multiple data sources, with synchronization errors controlled within 5-10ms.

[0061] Resource scheduler: Dynamically allocates CPU / NPU computing power based on vehicle operating conditions (low speed / high speed, idling / driving) and CAN bus load rate (sampling interval 100ms), prioritizing the real-time performance of AI modeling and voice separation; when the bus load is >60% (congested road conditions), it automatically reduces the computing power ratio of non-core modules (such as user feedback data statistics), reserving 15% redundant computing power for synchronous compensation and lightweight model inference.

[0062] Health Monitor: Real-time monitoring of sensor status (microphone failure, vibration sensor offline) and AI model output confidence (speech separation signal-to-noise ratio SDR is lower than a preset threshold), triggering fault warnings or model rollback; during rollback, it directly calls the most recent compliant operating condition parameters saved by EWC incremental learning to avoid loss of historical adaptation capabilities, with a rollback latency of ≤10ms.

[0063] 1.2 Sound Field Perception Module (Sensor blind spot filling, covering 3D sound field)

[0064] Responsible for collecting in-vehicle full-domain 3D sound field and environmental data to provide input for AI modeling, including:

[0065] Multimodal sensor array: It consists of a 4-8 channel "ring + vertical" microphone array (ring deployed on the roof interior and door pillars, and vertically deployed on the front seat headrests and rear roof) and 4 vibration sensors (attached to the engine compartment, left front wheel chassis, right front wheel chassis and rear chassis respectively); the microphone sampling rate is 48kHz and the bit depth is 24bit, and the vibration sensor sampling rate is 1kHz.

[0066] Data preprocessing unit: Performs DC removal, pre-emphasis (high frequency compensation) and frame segmentation on the raw acquired data to filter out sensor hardware noise; for the interpolated vibration data, an additional 5ms sliding window smoothing filter is added to avoid high frequency interference introduced by interpolation.

[0067] CAN data interface: Reads dynamic operating condition data (vehicle speed v, engine speed n, window opening θ, air conditioning fan speed w) and bus load rate from the vehicle CAN bus as dynamic environmental parameters for sound field modeling; when the bus load is >80%, a data caching mechanism is triggered (caching the most recent 5 frames of operating condition parameters) to avoid data loss.

[0068] 1.3 AI Sound Field Modeling Engine

[0069] The core innovative module of this invention runs on the AI ​​computing core of the main control unit, and realizes real-time dynamic modeling of the vehicle sound field through a lightweight multi-dimensional AI algorithm, including:

[0070] Sound field feature extraction engine: Extracts sound field features in the time domain (short-time energy, zero-crossing rate), frequency domain (STFT spectrum, Mel-frequency cepstral coefficients MFCC), and 3D spatial domain (horizontal + vertical TDOA, beamforming gain), and combines attention mechanism to screen key features; optimizes feature dimensions and reduces redundant calculations.

[0071] Dynamic sound field modeling engine: The sound field transfer function model is constructed using "lightweight STCN + bidirectional LSTM", which integrates vehicle operating parameters (vehicle speed v, engine speed n, window opening θ) and CAN bus load rate to update the 3D sound field spatial distribution in real time; both the STCN convolution kernel and LSTM structure are designed to be lightweight to reduce computing power requirements.

[0072] Noise classification and suppression engine: Based on the CNN-LSTM two-stage classification network, noise types (periodic engine noise, broadband wind noise, impulse road noise and two types of mixed noise) are identified. Adaptive suppression algorithms are designed for single and mixed noise respectively to solve the problem of residual mixed noise.

[0073] 1.4 AI Voice Separation Module

[0074] Based on 3D sound field models and noise suppression results, accurate separation of target speech (such as driver instructions and rear passenger conversations) from interfering noise is achieved, including:

[0075] Spatial Constraint Separation Unit: Utilizes the 3D spatial distribution information output by the sound field modeling engine to construct a 3D spatial mask of the target speech (including horizontal and pitch angle weights) to initially filter out noise from non-target directions; in rear-row speech scenarios, it automatically increases the vertical mask weights.

[0076] End-to-end separation network: An improved lightweight Conv-TasNet is adopted, which incorporates 3D sound field features and noise classification results (including mixed noise labels) and optimizes the separation weights; for mixed noise scenarios, a lightweight enhancement branch is automatically activated to supplement the separation capability.

[0077] Separation quality assessment unit: Real-time calculation of signal-to-noise ratio (SDR), source distortion ratio (SIR), and residual mixed noise rate (the proportion of non-target noise energy in the target frequency band) of the separated speech, as feedback indicators for model optimization; differentiated setting of accuracy thresholds (single noise SDR ≥ 12dB, mixed noise SDR ≥ 10dB).

[0078] 1.5 Real-time control unit

[0079] The system runtime parameters (non-structural parameters) are dynamically adjusted based on the speech separation effect, including:

[0080] Adaptive parameter adjuster: Based on the separation quality assessment results, optimize the microphone array gain and AI model runtime parameters (such as LSTM learning rate, attention mechanism weight coefficients, and noise suppression algorithm protection coefficients).

[0081] Feedback learning unit: Reinforcement learning (DQN) is used to construct a multi-objective reward function (SDR improvement, latency reduction, and mixed noise residual rate reduction) to continuously iterate and optimize the AI ​​model; the CAN bus load rate weight is introduced into the reward function (when the load is high, stability is prioritized and latency assessment weight is reduced).

[0082] 1.6 Human-Computer Interaction Module (Scene Adaptation, Enhancing User Experience)

[0083] The system outputs clear, separated voice messages, supports voice command recognition (interfacing with in-vehicle infotainment systems), call noise reduction (interfacing with Bluetooth phones), and provides a user feedback interface (such as a separation effect rating button). In high-speed wind noise scenarios, it automatically increases the suppression depth of the 5000-8000Hz frequency band for call noise reduction by 5dB. In rear-seat voice scenarios, it optimizes the vertical voice output gain.

[0084] The beneficial effects of this embodiment are as follows: by using multi-source data synchronization, 3D sound field dynamic modeling, and noise classification and suppression technology, it achieves accurate separation of voice and noise in in-vehicle scenarios, solving the problems of dynamic noise interference and sound field variation; by combining lightweight algorithms and real-time control, it balances separation accuracy (SDR≥12dB) and latency (≤100ms) to adapt to mixed noise scenarios; and by scene adaptation (such as rear-seat voice gain and high-speed wind noise suppression), it improves user experience and ensures the clarity and stability of in-vehicle voice interaction.

[0085] Example 2:

[0086] Sound field feature extraction engine (adapted to 3D synchronization and feature dimension)

[0087] For a detailed workflow of the sound field feature extraction engine, please refer to [link / reference]. Figure 3As shown, the core task is to extract features from multi-source sensing data that characterize the essence of the in-vehicle 3D sound field, providing a foundation for subsequent modeling and separation. The specific workflow is as follows:

[0088] 2.1 Multi-source data input and synchronization.

[0089] The engine first receives microphone audio data from the multimodal sensor array. Data from vibration sensors The timing is synchronized to a unified time axis (based on the microphone sampling clock) through a timing synchronizer to achieve time-dimensional correlation matching (not a "one-to-one correspondence at the same moment"); at the same time, the operating parameters of the CAN bus (vehicle speed v(t) at time t, engine speed n(t)) and bus load rate are read and marked as environmental feature vectors. ;

[0090] When the CAN data transmission delay is greater than 10ms, a linear prediction algorithm is used to supplement the estimated values ​​of the operating parameters at the current moment to ensure the continuity of timing.

[0091] 2.2 Multi-domain feature extraction (3D spatial feature extension, hardware adaptation)

[0092] Temporal feature extraction: Calculating short-time energy from framed audio data (Reflecting sound intensity) and zero-crossing rate (Reflecting the frequency of sound), the formula is as follows:

[0093] ;

[0094] ;

[0095] in, Reflects sound intensity (the higher the value, the higher the energy); It reflects the frequency characteristics of the signal (high-frequency signals have a higher zero-crossing rate); Let N be the kth sampling point of the t-th frame, and N be the frame length. For the interpolated vibration data, the same formula is used to calculate the short-time energy as a supplementary dimension of the temporal features.

[0096] Frequency domain feature extraction: Perform STFT on each frame of audio to obtain the spectrum, and calculate the 13-dimensional MFCC, as shown in the following formula:

[0097] ;in, This is the Mel filter bank, and DCT is the Discrete Cosine Transform, used to reduce dimensionality and preserve key frequency domain information; This is the spectrum after STFT.

[0098] For engine noise scenarios, additionally calculate the peak intensity of spectral harmonics (the fundamental frequency and the energy proportion of its 2nd to 5th harmonics) as a supplement to the frequency domain characteristics.

[0099] 3D Spatial Domain Feature Extraction: 3D Geometric Position Based on Microphone Array (Known Coordinates) ), calculate the horizontal + vertical time delay difference TDOA between any two microphones. 3D spatial gain is calculated by combining beamforming algorithms. ( For horizontal incident angle, (where the angle is pitch), the formula is as follows:

[0100] ;

[0101] ;

[0102] in, Coordinates; The speed of sound (340 m / s); Microphone weights (vertical microphone weights are dynamically adjusted based on the target position); Let TDOA be the difference between the i-th microphone and the reference microphone; Number of microphones; .

[0103] The spatial domain feature dimensions have been expanded from 8 to 12, covering the in-vehicle 3D sound field.

[0104] 2.3 Key Feature Filtering and Fusion (Weight Adaptation Scenarios)

[0105] The channel attention mechanism (ECA-Net) is used to assign weights to the extracted multi-domain features (4-dimensional time domain + 14-dimensional frequency domain + 12-dimensional spatial domain), as shown in the following formula:

[0106] ; ;in, The attention weight for the k-th feature is used to measure the importance of this type of feature in the key feature selection. It is calculated through the attention mechanism and will be used to perform weighted fusion of various features. Importance score of the k-th feature Performing exponential operations is for calculating attention weights. The molecular part is used to highlight the differences in importance scores of different features; Importance scores for all M-class features The attention weights are calculated by performing exponential operations on each item and then summing the results. The denominator is used to normalize the indexed importance scores of various features, making... The sum of is 1; For the k-th type of feature; The score represents the feature importance score; M=30 represents the total feature dimension. The filtered key feature vectors are output to the dynamic sound field modeling engine; at the same time, feature importance logs are saved for subsequent optimization of the filtering strategy.

[0107] The beneficial effects of this embodiment are as follows: by multi-source synchronization, full-dimensional feature extraction and attention filtering, the comprehensiveness and computational efficiency of vehicle sound field feature representation are improved, laying the foundation for dynamic sound field modeling and speech separation.

[0108] Example 3:

[0109] Dynamic sound field modeling engine (lightweight and incremental learning synergy)

[0110] The core of the dynamic sound field modeling engine is to construct a 3D sound field transfer function model that changes in real time with the vehicle's operating conditions, accurately depicting the propagation patterns of sound in the vehicle space. The specific working principle is as follows:

[0111] 3.1 Sound Field Modeling Objectives and Inputs

[0112] The modeling goal is to construct any 3D position within the vehicle space. sound pressure level of the sound field The mapping relationship with sound sources (engine, external noise, target speech),

[0113] Input includes:

[0114] 30-dimensional key features output by the sound field feature extraction engine ;

[0115] Vehicle operating parameters (θ is the window opening) and CAN bus load rate;

[0116] Preset 3D model of in-vehicle space (seat position, 3D sound absorption coefficient of interior materials, including the vertical sound absorption coefficient of roof fabric and floor carpet).

[0117] 3.2 Spatiotemporal Dynamic Modeling Network Architecture

[0118] The dynamic sound field model is constructed using a lightweight STCN + bidirectional LSTM approach, divided into two stages, reducing total computing power by 40%.

[0119] In the spatial modeling stage: the STCN convolution kernel was optimized to "2×2×2 spatial convolution (covering x / y / z 3D) + 1×1 temporal convolution", reducing the number of convolution kernels from 64 to 48. This captures the spatial distribution differences of the 3D sound field (such as the sound pressure difference between the driver's ear and the rear child's ear) and incorporates the 3D sound absorption coefficient of the interior materials. (such as seat fabric) ,Glass The formula is as follows: ;in, 3D spatial sound field feature map; These are the key feature vectors after filtering.

[0120] In the time-dynamic modeling stage: the bidirectional LSTM captures the temporal change trend of the sound field (such as the gradual change of wind noise when the vehicle speed increases). The input is a fusion vector of "key features + operating parameters + load rate" (30+4+1=35 dimensions). The LSTM hidden layer dimension is fixed at 256 dimensions, and the output is the 3D sound field transfer function at time t. ;in, For frequency (20Hz-20kHz); Represents frequency Below, sound source to 3D position The sound pressure transmission coefficient.

[0121] 3.3 Model Incremental Update Mechanism (in conjunction with rollback)

[0122] Considering the dynamic changes in in-vehicle operating conditions (such as windows changing from closed to open), an incremental learning algorithm (EWC elastic weight consolidation) is used to update model parameters to avoid latency caused by retraining. The update formula is as follows: ;in, These are model parameters; The learning rate (dynamically adjusted according to the CAN bus load rate: 0.01 when load rate < 50%, 0.005 when load rate > 80%). The loss function is the mean squared error (MSE). (Measured sound pressure level) The regularization coefficient is used. ᵢ represents the parameter importance matrix; The parameters are the best historical values ​​(linked with the rollback mechanism of the health monitor to periodically save the parameters that meet the criteria). The sound pressure level of the sound field output by the model at the current moment is the predicted value of the 3D sound field sound pressure level of the vehicle space by the dynamic sound field modeling engine at time t. Let the loss function L be the output of the current model. The gradient; This indicates gradient calculation, which determines the direction of parameter updates by calculating the gradient, so that the model is adjusted in the direction of reducing error.

[0123] The logic for adaptive update frequency is as follows: When the operating conditions are stable, the update interval is extended from 100ms to 500ms to reduce computing power consumption.

[0124] 3.4 Verification of Modeling Accuracy

[0125] Real-time measurement of sound pressure levels at five fixed test points in the vehicle interior (near the driver's ear, near the front passenger's ear, left rear seat, middle rear seat, and right rear seat). , and the model output ( , It is the 3D sound field transfer function. The error is calculated as the sound source intensity, and the error is required to be ≤3dB. If it exceeds the threshold, the EWC regularization coefficient is finely adjusted first to reduce parameter fluctuations. At the same time, the working conditions where the error exceeds the standard (such as open car window + high speed) are recorded for subsequent offline optimization training data.

[0126] The beneficial effects of this embodiment are: through lightweight architecture, incremental learning and adaptive updating, high-precision and low-latency modeling of vehicle dynamic sound field is achieved, balancing computational efficiency and model adaptability.

[0127] Example 4:

[0128] Noise Classification and Suppression Engine

[0129] The core of the noise classification and suppression engine is to accurately identify single / mixed vehicle noise and suppress it accordingly, reducing interference for speech separation. The specific working principle is as follows:

[0130] 4.1 Noise Data Acquisition and Labeling

[0131] First, a vehicle noise dataset was constructed, covering three types of single noise and two typical mixed noise types, with the sample proportions as follows: single noise 70%, mixed noise 30%.

[0132] Periodic noise: generated by engine operation, frequency 200-2000Hz, varies with engine speed n (fundamental frequency 10Hz when n=600rpm, fundamental frequency 50Hz when n=3000rpm).

[0133] Broadband noise: wind noise (significant at v≥60km / h, frequency 500-5000Hz), road noise (frequency 100-2000Hz, varying with road surface roughness);

[0134] Impulse noise: generated by tires going over speed bumps and stones hitting the chassis, lasting ≤100ms, with a high peak sound pressure level;

[0135] Mixed noise 1: Engine noise + wind noise (vehicle speed 80-120km / h + engine speed 2000-3000rpm);

[0136] Mixed noise 2: Road noise + impulse noise (rough road surface + speed bumps);

[0137] The collected noise data are labeled with "noise type - operating condition parameters - CAN bus load rate" to construct training and validation sets.

[0138] 4.2 Training and Inference of Noise Classification Network

[0139] A two-stage classification network, CNN-LSTM, is used for mixed noise recognition.

[0140] CNN Feature Extraction: Use 3 convolutional layers (5×5 kernel, stride 1) + 1 depthwise separable convolutional layer (reducing computational power by 25%) to extract local features of the noise spectrum (such as harmonic peaks of engine noise and broadband spectrum of wind noise).

[0141] LSTM temporal modeling: Using a 2-layer LSTM to capture the temporal variation of noise (such as the fundamental frequency increase during engine acceleration and the short-term burst characteristics of impulse noise);

[0142] Classification output: The probability P of 5 types of noise is output by a fully connected layer + Softmax: ( For periodic noise probability, Let be the broadband noise probability. This represents the probability of impulse noise. Given mixed noise with a probability of 1, (For mixed noise with two probabilities), the category with the highest probability is taken as the classification result. .

[0143] Judgment rules: If the maximum probability is >85%, it is single noise; if the maximum probability is ≤85% and the probability difference between any two types of noise is <15%, it is the corresponding mixed noise.

[0144] The loss function used in classification networks is cross-entropy loss. ;in, For labeled one-hot vectors (mixed noise is labeled with two labels, such as mixed noise 1 labeled as "engine noise + wind noise"), the classification accuracy after training should be ≥95%; The predicted probability of the c-th type of noise output by the classification network is obtained after Softmax, reflecting the confidence level of the model in classifying it as this type of noise.

[0145] 4.3 Targeted Noise Suppression

[0146] Select the suppression algorithm based on the classification results:

[0147] like (Engine noise): An adaptive notch filter (ANF) is used, combined with vibration sensor data to assist in FFT spectral peak detection. The notch frequency is designed to be... (Based on FFT spectral peak detection), the notch filter frequency is designed to be... , , , , (Covering 1st-5th harmonics, suitable for multi-cylinder engines) Suppression depth ≥20dB;

[0148] like (Wind noise / Road noise): Noise spectrum estimation is performed using an improved spectral subtraction method combined with the output of the sound field model. Differentiated protection coefficient settings: ;in, The spectrum of the noisy signal; To protect the coefficient and avoid speech distortion, Wind noise scenario: (High-frequency noise accounts for a large proportion, requiring the preservation of more speech details); Road noise scenario: Primarily low-frequency noise; noise can be moderately compressed. (Engineering default value:) (Compatible with most in-vehicle scenarios); Frequency dimension; The time dimension; The calculation method is as follows: Among them, STCN is a spatiotemporal convolutional network that outputs a 3D sound field spatial distribution; For noise source spectrum templates (such as engine noise fundamental harmonics, wind noise broadband spectrum).

[0149] like The detection method uses "vibration sensor synchronous triggering + short-time energy threshold" (threshold = 2 times steady-state energy). When a pulse is detected, the pulse frame is replaced by linear interpolation of the preceding and following frames to preserve the continuity of speech.

[0150] like The parallel processing of "ANF (engine noise suppression) + improved spectral subtraction (wind noise suppression)" is adopted, and the ANF output is directly input into the spectral subtraction to avoid cascade delay.

[0151] like The method employs "pulse detection and replacement (processing pulses first) + improved spectral subtraction (suppressing road noise later)" to avoid interference from pulses on spectral subtraction.

[0152] The beneficial effects of this embodiment are as follows: by constructing an in-vehicle dataset covering single / mixed noise (the sample ratio matches the actual scene), and combining it with a CNN-LSTM classification network, five types of noise (including mixed noise) are accurately identified; targeted suppression algorithms (ANF, improved spectral subtraction, impulse detection replacement) are adapted to different noise characteristics, with a suppression depth of ≥20dB and avoiding speech distortion; mixed noise is processed in parallel / cascaded to reduce cascaded delay, effectively reduce the interference of noise on speech separation, and improve the clarity of in-vehicle speech.

[0153] Example 5:

[0154] AI voice separation module

[0155] The AI ​​speech separation module is the core functional module. Based on the 3D sound field model and noise suppression results, it achieves accurate separation of the target speech. The specific working principle is as follows:

[0156] 5.1 Decoupled Network Architecture Design (Lightweight and Branch Adaptation)

[0157] An improved, lightweight Conv-TasNet is adopted, incorporating in-vehicle 3D sound field and noise type information. The network is divided into three parts, reducing computing power by 30%.

[0158] Encoder: 1-layer 1D convolution (kernel size 3, stride 1, number of kernels reduced from 128 to 96) to convert noisy speech (after noise suppression) into feature maps;

[0159] Separate network: 6 basic temporal convolutional blocks (TCBs), each TCB containing 1×1 convolutions (256 channels) and causal dilated convolutions (dilation rate). (k=0-5) and 3D spatial attention layer (3D spatial mask integrated into the sound field model) In mixed noise scenarios, automatically activate a 2-layer lightweight TCB (128 channels) to output a separation mask. (Target speech mask) and (Residual noise mask);

[0160] Decoder: A single transposed convolutional layer restores the separated feature maps to a time-domain speech signal. (Target speech) and (Residual noise) The transposed convolution stride is kept consistent with the encoder to avoid timing misalignment.

[0161] 5.2 Spatial Constraints and Loss Functions (Multi-Objective Adaptation)

[0162] To improve separation accuracy, two constraints are introduced to adapt to mixed noise scenarios:

[0163] 3D spatial constraints: based on the 3D location of the target speech (e.g., driver) Rear passengers Extract the 3D spatial gain at that location from the sound field model. ,Will The spatial weights are integrated into the separation network as attention weights; in mixed noise scenarios, the spatial weights are extended to a 5cm range around the target location, covering the spatial diffusion characteristics of speech propagation.

[0164] Multi-task loss: The loss function includes time-domain MSE loss, frequency-domain SDR loss, and mixed noise residual loss, with differentiated weight settings: ;in, This is a multi-task loss function used to measure the difference between the speech separation result and the clean target speech. The training objective is to minimize this loss. These are weighting coefficients in mixed noise scenarios. This is used to differentiate the weights of time-domain MSE loss and frequency-domain SDR loss, thereby enhancing residual noise suppression. The separated target speech signal (time domain); For clean target speech (annotated data, i.e., the ideal noise-free target speech time-domain signal); The mean square error (MSE) loss in the time domain measures the error between the separated speech and the clean target speech in the time domain. Source distortion ratio (SDR) is used to evaluate the quality of separated speech relative to clean target speech. It reflects the degree of separation between target speech and interference. The higher the value, the better the separation effect. As a frequency domain loss term, it together with the time domain MSE loss constitutes the multi-task loss.

[0165] 5.3 Real-time optimization and separation quality assessment (flow process and index adaptation)

[0166] To meet the real-time requirements of in-vehicle systems, three optimizations are adopted to align with the computational logic:

[0167] Model compression: The separated network was subjected to INT8 quantization (quantization error controlled within ±0.5dB in the core speech frequency band 200-3400Hz) and structured pruning, reducing the model size from 200MB to 60MB.

[0168] Pipeline processing: Implement pipelined operations on automotive GPUs (such as NVIDIA Orin) where the encoder processes frame t and the split network processes frame t-1 (avoiding serial dependencies) to reduce latency;

[0169] Dynamic computing power allocation: In single-noise scenarios, two layers of lightweight TCBs are disabled; in mixed-noise scenarios, all TCBs are activated to balance accuracy and latency; the separation quality assessment unit calculates the SDR, SI-SNR, and mixed noise residual rate of the separated speech in real time, and sets thresholds differently.

[0170] Single noise scenario: SDR≥12dB, SI-SNR≥10dB;

[0171] Mixed noise scenario: SDR≥10dB, SI-SNR≥8dB, residual rate≤15%; if below the threshold, trigger the real-time control unit to optimize runtime parameters (such as spatial attention weight, spectral subtraction protection coefficient).

[0172] The beneficial effects of this embodiment are as follows: by using a lightweight Conv-TasNet architecture and model compression, combined with pipelined processing, it meets the real-time requirements of in-vehicle systems; 3D spatial constraints cover the spatial diffusion characteristics of speech propagation, and the multi-task loss function adapts to mixed noise scenarios, improving separation accuracy; dynamic computing power allocation balances accuracy and latency, optimizing user experience.

[0173] Example 6:

[0174] The method of this invention includes the following steps, realizing real-time processing of the entire process from sound field perception to speech separation, and adapting to multi-source data synchronization, mixed noise, and vehicle hardware characteristics:

[0175] Step S101: Real-time acquisition and synchronization of multi-source data (adapting to differences in sampling frequency)

[0176] The multimodal sensor array collects microphone audio data and vibration sensor data; the CAN data interface reads vehicle operating parameters (v, n, θ) and bus load rate; the timing synchronizer uses the microphone sampling clock as a reference and aligns the multi-source data to the same time axis through "PPS synchronization + dynamic compensation", with a timestamp error ≤10ms, and outputs synchronized data D(t) (including audio, interpolated vibration data, operating parameters, and load rate).

[0177] Step S102: Extraction of key sound field features (3D feature and weight matching)

[0178] The sound field feature extraction engine divides the synchronous data D(t) into frames (20ms frame length, 10ms frame shift), extracts features in the time domain (4D), frequency domain (14D), and 3D spatial domain (12D), and filters 30 key features through the ECA-Net attention mechanism (scene-based weights) and outputs them to the AI ​​sound field modeling engine.

[0179] Step S103: Dynamic sound field modeling and updating (lightweighting and rollback coordination)

[0180] The dynamic sound field modeling engine inputs key features, operating parameters, and load rate into a "lightweight STCN + bidirectional LSTM" network to construct a 3D sound field transfer function H(x,y,z,f,t); it adaptively updates the model parameters through EWC incremental learning (extending the update interval under stable operating conditions); it collects measured sound pressure levels at 5 test points to verify accuracy (error ≤3dB) and outputs the sound field model M(t); if the error exceeds the standard, it fine-tunes the EWC regularization coefficient.

[0181] Step S104: Noise Classification and Targeted Suppression (Combined Processing of Mixed Noise)

[0182] The noise classification and suppression engine inputs noisy speech into a CNN-LSTM classification network, outputs five noise probabilities and determines the type; then, it selects a suppression algorithm based on the type.

[0183] A single algorithm is used for single types of noise, and a joint algorithm is used for mixed noise, outputting the denoised speech signal. With noise type identifier C(t).

[0184] Step S105: AI Speech Separation and Quality Assessment (Pipeline and Differentiated Metrics)

[0185] The AI ​​speech separation module inputs the denoised speech, sound field model M(t), and noise type identifier into a lightweight Conv-TasNet: single noise activates 6 layers of TCB, mixed noise activates all TCB, incorporates 3D spatial constraints and multi-task loss, and outputs the target speech; the separation quality assessment unit calculates SDR, SI-SNR and residual rate, and outputs the assessment results.

[0186] Step S106: Real-time control and feedback (rollback and parameter adjustment)

[0187] If the scenario-specific criteria are met (single noise SDR ≥ 12dB, mixed noise SDR ≥ 10dB and residual rate ≤ 15%), the target speech is output to the human-computer interaction module, and the output parameters are adaptively adjusted according to the scenario.

[0188] If the requirements are not met, the real-time control unit optimizes runtime parameters such as microphone gain and attention weight through reinforcement learning and feeds them back to S102 (feature selection weight), S104 (suppression algorithm parameters), and S105 (TCB activation state). The health monitor simultaneously monitors the sensor and model status. If an alarm is triggered, it calls the historical compliant parameters to roll back the model to ensure stable system operation.

[0189] The beneficial effects of this embodiment are as follows: through full-process collaborative design, high precision and low latency of in-vehicle voice separation are achieved; multi-source data synchronization (error ≤10ms) ensures input reliability; 3D sound field features and mixed noise classification and suppression reduce interference; lightweight Conv-TasNet (latency ≤50ms) combined with spatial constraints and multi-task loss adapts to single / mixed noise scenarios (SDR≥10dB); real-time adjustment and model rollback mechanism ensure system stability, solve the technical problems of dynamic noise and changing sound field in in-vehicle scenarios, and improve the clarity of voice interaction and user experience.

[0190] All formulas in this invention are dimensionless and calculated numerically. The preset parameters in the formulas can be set by those skilled in the art according to the actual situation.

[0191] The weighting coefficients of this invention are used to measure the degree of influence of different factors or variables on a certain outcome or decision. The weighting coefficient is defined as the numerical value assigned to each factor when comparing and evaluating multiple factors, reflecting their importance or priority. These weighting coefficients can be determined according to specific circumstances and needs, and are usually jointly formulated and confirmed by professionals or relevant stakeholders. By reasonably setting the weighting coefficients, programs or systems can be helped to make decisions or predictions more accurately.

[0192] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0193] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. An AI-driven real-time modeling of in-vehicle sound field and speech separation method, characterized in that, The specific steps are as follows: Step S101: The multi-modal sensor array collects microphone audio data, which is supplemented to the vibration sensor data through linear interpolation; the CAN data interface reads the vehicle operating condition parameters and bus load rate; the time sequence synchronizer synchronizes the multi-source data based on the microphone sampling clock through PPS synchronization + dynamic compensation, so that the timestamp error does not exceed the set threshold, and outputs the synchronized data; Step S102: The sound field feature extraction engine extracts time domain, frequency domain and 3D spatial domain features from the synchronized data, filters 30-dimensional key features through the ECA-Net attention mechanism, and outputs to the AI sound field modeling engine; Step S103: The dynamic sound field modeling engine inputs the key features, operating condition parameters and load rate into the lightweight STCN + bidirectional LSTM network to construct a 3D sound field transfer function; the EWC incremental learning is used to adaptively update the model parameters, stabilize the operating conditions and prolong the update interval; the sound pressure level of five test points is measured to verify the precision, and the sound field model is output, and the EWC regularization coefficient is fine-tuned when the error exceeds the threshold; Step S104: The noise classification and suppression engine inputs the noisy speech into the CNN-LSTM classification network, outputs the probability of five types of noise and determines the type; According to the type, the suppression algorithm is selected, and the denoised speech signal and noise type identifier are output; Step S105: The AI speech separation module inputs the denoised speech, sound field model and noise type identifier into the lightweight Conv-TasNet, integrates 3D spatial constraints and multi-task loss to output the target speech; the separation quality evaluation unit calculates SDR, SI-SNR and residual rate and outputs the results; Step S106: When the scene-based indicators are met, the target speech is output to the human-computer interaction module and the output parameters are adaptively adjusted; otherwise, the real-time control unit optimizes the runtime parameters through reinforcement learning and feeds back to S102, S104 and S105, and the health monitor monitors the state and triggers the warning to call the historical qualified parameter rollback model.

2. The AI-driven real-time modeling of in-vehicle sound field and speech separation system for implementing the AI-driven real-time modeling of in-vehicle sound field and speech separation method as claimed in claim 1, characterized in that, The system comprises a main control unit, a sound field perception module, an AI sound field modeling engine, an AI speech separation module, a real-time control unit and a human-computer interaction module; The main control unit: as the central dispatcher of the system, deployed on the vehicle domain controller, with low-delay computing and multi-module collaboration capability, responsible for the whole process management of sound field perception, AI modeling, speech separation and real-time control, and containing three sub-components of time sequence synchronizer, resource scheduler and health monitor; The sound field perception module: responsible for collecting 3D sound field data and environmental data in the vehicle environment, providing input for AI modeling, including a multi-modal sensor array composed of a microphone array and a vibration sensor for collecting audio and vibration information; The data preprocessing unit is used for cleaning the original data, and the CAN data interface is used for acquiring dynamic environmental parameters; The AI sound field modeling engine: realizes real-time dynamic modeling of the vehicle sound field through lightweight multi-dimensional AI algorithm, and is used for sound field feature extraction, dynamic sound field modeling and noise classification and suppression; AI voice separation module: based on 3D sound field model and noise suppression results, realize the accurate separation of target voice and interference noise, use spatial constraint separation unit, end-to-end separation network and separation quality evaluation unit to improve separation precision and reduce delay; Real-time regulation unit: dynamically adjust system running parameters according to voice separation effect to ensure optimal performance, including adaptive parameter regulator for optimizing microphone array gain and AI model running time parameters; Feedback learning unit uses reinforcement learning method to continuously optimize AI model; Human-computer interaction module: output voice, support voice instruction recognition and conversation noise reduction, and provide user feedback interface; automatically adjust parameters according to scene.

3. The AI-driven on-board sound field real-time modeling and speech separation system of claim 2, wherein, The main control unit is deployed in the vehicle-mounted domain controller, has low delay calculation and multi-module cooperation capability, and internally includes: Timing synchronizer: synchronize microphone array, vibration sensor and vehicle-mounted CAN bus data through hardware PPS pulse synchronization signal + software dynamic compensation algorithm, for the difference of multi-source data sampling / update frequency, use time axis alignment + data interpolation matching strategy to linearly interpolate 1kHz original data of vibration sensor to 48kHz time sequence, and compensate the transmission delay of CAN data through the first 3 frames of parameters to ensure the consistency of multi-source data timestamp, and the synchronization error is controlled within the set threshold; Resource scheduler: dynamically allocate CPU / NPU computing power according to vehicle working condition and CAN bus load rate, adjust CPU / NPU computing power dynamically to ensure that the overall system delay does not exceed the set time; in congested road conditions, automatically reduce the proportion of non-core module computing power, and reserve a certain proportion of redundant computing power for synchronous compensation and model lightweight inference; Health monitor: real-time monitoring of sensor state and AI model output confidence, triggering fault warning or model rollback; when rolling back, directly call the last time standard working condition parameters saved by EWC incremental learning, and the rollback delay does not exceed the set time.

4. The AI-driven on-board sound field real-time modeling and speech separation system of claim 2, wherein, The sound field perception module is responsible for collecting vehicle-mounted global 3D sound field and environmental data, including: Multi-modal sensor array: composed of 4-8 ring + vertical microphone array and 4 vibration sensors; Data preprocessing unit: remove direct current, pre-emphasize and frame process the original collected data to filter out sensor hardware noise; for interpolated vibration data, additional 5ms sliding window smoothing filter is added to avoid high frequency interference introduced by interpolation; CAN data interface: read dynamic working condition data and bus load rate of vehicle-mounted CAN bus as dynamic environmental parameters for sound field modeling; when the bus load is greater than or equal to the set proportion, trigger the data caching mechanism to avoid data loss.

5. The AI-driven on-board sound field real-time modeling and speech separation system of claim 2, wherein, The AI sound field modeling engine runs in the AI computing core of the main control unit, including: Sound field feature extraction engine: extract time domain, frequency domain and 3D spatial domain features of sound field, and select key features combined with attention mechanism; optimize feature dimension to reduce redundant calculation; Dynamic sound field modeling engine: a lightweight STCN + bidirectional LSTM is used to construct a sound field transfer function model, which combines vehicle operating parameters and CAN bus load rate to update the 3D sound field spatial distribution in real time; the STCN convolution kernel and LSTM structure are both designed to be lightweight to reduce the demand for computing power; Noise classification and suppression engine: a CNN-LSTM two-stage classification network is used to identify noise types, and adaptive suppression algorithms are designed for single / mixed noise.

6. The AI-driven on-board sound field real-time modeling and speech separation system of claim 2, wherein, The AI voice separation module is based on a 3D sound field model and noise suppression results to achieve accurate separation of target voice, including: Spatial constraint separation unit: use the 3D spatial distribution information output by the sound field modeling engine to construct a 3D spatial mask for the target voice, and preliminarily filter non-target direction noise; in the rear voice scene, automatically increase the vertical direction mask to the set weight; End-to-end separation network: use an improved lightweight Conv-TasNet, integrate 3D sound field features and noise classification results, and optimize the separation weight; for mixed noise scenes, automatically activate the lightweight enhanced branch to supplement the separation capability; Separation quality evaluation unit: real-time calculation of the signal-to-noise ratio, source distortion ratio and mixed noise residual rate of the separated voice as feedback indicators for model optimization; set different accuracy thresholds. In the improved lightweight Conv-TasNet of the AI voice separation module, the encoder is a 1D convolution, the separation network is a 6-layer basic time convolution block, the 2-layer lightweight TCB is automatically activated in the mixed noise scene, and the decoder is a 1-layer transpose convolution; 3D spatial constraints and multi-task loss are introduced; through model compression, pipeline processing and dynamic power allocation, the real-time requirements of vehicle-mounted systems are met, and the separation quality evaluation thresholds are set differently.

7. The AI-driven on-board sound field real-time modeling and speech separation system of claim 2, wherein, The real-time regulation unit dynamically adjusts the system runtime parameters according to the voice separation effect, including: Adaptive parameter adjuster: based on the separation quality evaluation results, optimize the microphone array gain and AI model runtime parameters; Feedback learning unit: use reinforcement learning to continuously iterate and optimize the AI model; introduce CAN bus load rate weight in the reward function.

8. The AI-driven on-board sound field real-time modeling and speech separation system of claim 2, wherein, The human-computer interaction module outputs the clear voice after separation, supports voice command recognition, call noise reduction, and provides user feedback interface; in high-speed wind noise scenes, automatically increase the set frequency band suppression and set depth of call noise reduction; in the rear voice scene, optimize the vertical direction voice output gain.

9. The AI-driven on-board sound field real-time modeling and speech separation system of claim 5, wherein, The noise classification and suppression engine constructs a vehicle-mounted dataset covering 3 types of single noise and 2 types of typical mixed noise, with a sample proportion of 70% single noise and 30% mixed noise; a CNN-LSTM two-stage classification network is used, 3 convolution layers + 1 depth separable convolution are used for CNN feature extraction to extract noise spectrum local features, 2 LSTM layers are used for time series modeling to capture noise time variation rules, the classification output is output through a fully connected layer + Softmax to output 5 noise probabilities, and the type is determined according to the determination rule; the classification network loss function uses cross-entropy loss, and the classification accuracy after training is required to be greater than or equal to the set proportion; and the suppression algorithm is selected according to the classification results.

Images