A vehicle-mounted sound recognition and positioning intelligent early warning system and method

Abstract

The application discloses a kind of vehicle-mounted sound identification and positioning intelligent early warning system and method, it is related to vehicle-mounted intelligent early warning technical field.The system includes the audio signal of surrounding environment in real time acquisition by sound detector module, and sound identification module passes through deep learning model analysis audio signal and identifies sound source type.Calibration module carries out space-time calibration, ensure that each sensor data synchronization and alignment.Positioning module is combined to time difference of arrival TDOA and sound intensity attenuation model, through particle swarm optimization PSO and extended Kalman filter EKF technology optimization sound source positioning.Risk assessment module is based on sound source position, shielding state and object type assesses collision risk and promptly warns driver through early warning module, automatically triggers deceleration instruction.Self-learning module updates model through federation learning framework and dynamically adjusts early warning sensitivity and noise compensation coefficient.The application improves the environmental adaptability of vehicle-mounted early warning system, has significant safety improvement and wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle-mounted intelligent early warning, and in particular to a vehicle-mounted sound recognition and positioning intelligent early warning system and method. BACKGROUND

[0002] With the rapid development of intelligent driving technology, automatic driving systems and assisted driving systems have become an indispensable part in modern transportation. Vehicle-mounted cameras, radars, and LiDARs, etc. sensors play an important role in these systems, providing visual information and object detection capabilities for vehicles about the surrounding environment. However, existing systems still have significant limitations in certain specific environments and complex scenarios, especially in situations where the line of sight is limited, sensors are blocked, or environmental noise interference occurs. The detection capabilities of these traditional sensors are often severely affected, resulting in the system being unable to timely and accurately detect potential obstacles or hazards.

[0003] For example, in narrow urban roads, intersections, or situations where other vehicles block the view, the field of view of vehicle-mounted cameras and radars may be partially blocked, making it difficult to effectively detect pedestrians, bicycles, tricycles, or other obstacles in front of the vehicle. This "ghost probe" phenomenon is particularly prone to occur, leading to the vehicle being unable to make timely warning or avoidance actions during driving, thereby increasing the risk of traffic accidents. Traditional vehicle-mounted early warning systems usually rely on visual sensors and radars to assess collision risks by calculating the relative position between objects and vehicles. However, these systems have significant shortcomings in complex or blocked environments, especially when there are potential high-risk obstacles in front of the vehicle. Existing systems have difficulty providing accurate early warning information. In addition, existing systems also face the problem of environmental noise interference, especially in noisy environments such as urban streets and busy intersections. The signals collected by sensors may be disturbed, causing the system to fail to correctly identify and locate obstacles, thereby affecting the accuracy and timeliness of the warning.

[0004] Furthermore, the warning accuracy of the system is often limited by the performance of a single sensor, especially in situations where the detection area of the vehicle-mounted camera or radar is limited. This can result in "blind spots" where potential dangers are not identified in time. Therefore, how to achieve a higher precision and faster response early warning system in complex environments has become a key challenge in current technology development. SUMMARY

[0005] The application aims to provide a vehicle-mounted sound recognition and positioning intelligent early warning system and method, solve the problem that vehicle-mounted cameras and radars cannot accurately and timely identify and locate obstacles in complex environments such as shielding, poor line of sight, or environmental noise interference in the prior art, thereby significantly improving the response speed and accuracy of the vehicle-mounted intelligent early warning system in variable and complex driving environments, enhancing the environmental adaptability of the system, and effectively reducing the risk of traffic accidents.

[0006] In order to achieve the above-mentioned purpose, the application is realized through the following technical solutions:

[0007] A vehicle-mounted sound recognition and positioning intelligent early warning system comprises:

[0008] A sound detector module is configured to collect audio signals of the surrounding environment in real time.

[0009] A sound recognition module is configured to analyze the collected audio signals through a deep learning model, identify the type of sound source, and determine whether the sound source is shielded in combination with camera and radar data.

[0010] A calibration module is configured to precisely calibrate the audio signals with camera and radar detection data, optimize signal alignment through time synchronization and space mapping, and determine whether the sound source object is shielded.

[0011] A positioning module is configured to calculate the direction of the sound source object through the time difference of arrival (TDOA) of multiple sound detector signals, calculate the distance of the sound source object in combination with a sound intensity attenuation model and a particle swarm optimization (PSO) algorithm, correct the positioning result through extended Kalman filtering (EKF) fusion of camera and radar data, and dynamically adjust the noise compensation coefficient according to the environmental noise intensity.

[0012] A risk assessment module is configured to assess the collision risk based on the spatial position, shielding state, object type, and distance of the sound source, and generate early warning signals of different levels.

[0013] An early warning module is configured to display the sound source position on the vehicle-mounted display screen and issue a voice prompt, and trigger an automatic deceleration instruction when the collision risk meets the preset conditions.

[0014] A self-learning module is configured to update the model through a federated learning framework and dynamically adjust the early warning sensitivity and noise compensation coefficient according to the region type.

[0015] As a preferred scheme of the application, the sound recognition module comprises:

[0016] The timbre feature extraction unit extracts the feature vector of the audio signal through a mel-frequency cepstral coefficient (MFCC) and linear predictive coding (LPC) fusion algorithm. The fusion algorithm combines the feature vectors of MFCC and LPC through a weighted splicing method. The specific steps are as follows:

[0017] a. Preprocessing the audio signal, including denoising and framing operation;

[0018] b. Extracting the spectral features of the audio signal using the mel-frequency cepstral coefficient (MFCC) algorithm to obtain the feature vector:

[0019] MFCC = [mfcc1, mfcc2,..., mfcc N ];

[0020] Where: mfcc N represents the Nth feature value of the mel-frequency cepstral coefficient; N is the dimension of the MFCC feature;

[0021] c. Extracting the prediction coefficient features of the audio signal using the linear predictive coding (LPC) algorithm to obtain the feature vector:

[0022] LPC = [lpc1, lpc2,..., lpc M ];

[0023] Where: lpc M represents the Mth feature value of the LPC feature; M is the dimension of the LPC feature;

[0024] d. Fusing the MFCC and LPC feature vectors through a weighted splicing method to obtain the fused feature vector:

[0025] F fusion = [mfcc1, mfcc2,..., mfcc N , lpc1, lpc2,..., lpc M ];

[0026] Where: F fusion is the final fused feature vector, N and M are the dimensions of the MFCC and LPC features, respectively. In the weighted splicing, the weight coefficients of MFCC and LPC are adjusted according to the application environment or training data;

[0027] The classification unit is used to classify the fused audio feature vector based on the deep residual network (ResNet) model and output the source type label, including pedestrians, three-wheeled vehicles, motorcycles and other motor vehicles. The classification method is suitable for variable and complex noise environments through deep learning models;

[0028] The shielding judgment unit is used for judging that the sound source object is shielded when the vehicle-mounted camera or the radar fails to detect the target in the predicted sound source area, and the shielding judgment method improves the traditional shielding detection based on the visual sensor and combines the fusion of the sound signal and the visual signal.

[0029] As a preferred scheme of the present application, the calibration module comprises:

[0030] The time synchronization unit: the time of each sensor is calibrated by using the precision time protocol (PTP), and the synchronization error Δt≤1 ms;

[0031] The space mapping unit: the angle of the sound source position is mapped with the angle of the field of view of the vehicle-mounted camera, and the mapping formula is:

[0032]

[0033] Wherein θ is the horizontal angle of the sound source relative to the vehicle-mounted camera, φ is the vertical angle of the sound source relative to the vehicle-mounted camera, X source and Y source are the lateral and longitudinal coordinates of the sound source; and D is the distance between the sound source and the vehicle-mounted camera.

[0034] The shielding detection unit: if the vehicle-mounted camera or the radar fails to detect the target in the sound source prediction area, and the sound source category is a person or other motor vehicle, the source is marked as a shielded target.

[0035] As a preferred scheme of the present application, the positioning module comprises:

[0036] The azimuth calculation unit: the time delay difference between the sound detectors is calculated by using the generalized cross-correlation phase transform (GCC-PHAT) algorithm.

[0037] The distance estimation unit: the sound source distance is calculated by using the particle swarm optimization (PSO) algorithm based on the sound intensity attenuation model I(d) = I0-20log 10 (d)-αd+ò, wherein I(d) represents the signal intensity at a distance d; I0represents the signal intensity at a reference distance; d represents the distance between the sound source and the detector; α represents the sound intensity attenuation coefficient, which is usually related to the propagation environment and the frequency; and ò represents a noise term, which is a random error.

[0038] The position correction unit: when the camera / radar detects that the target is not shielded, the position calculation is optimized by fusing multi-source data through the extended Kalman filter (EKF).

[0039] The environmental noise compensation unit: the environmental noise intensity is monitored in real time, and the noise source term is dynamically adjusted, and the noise model is σ~N(0,σ 2 ), wherein σ represents the noise standard deviation; and N(0,σ 2) represents a Gaussian distribution with mean 0 and variance σ 2 representing environmental noise.

[0040] As a preferred scheme of the present application, the risk assessment module comprises:

[0041] The risk grading unit sets three early warning thresholds, and classifies according to the distance d: near d≤5m; medium 5

[0042]

[0043] wherein d0=10m, k=0.5m -1 , indicating the relationship between distance and alarm intensity;

[0044] The trajectory prediction unit predicts the future T=2s sound source position based on the long short-term memory network LSTM, and predicts the future motion trajectory of the sound source;

[0045] The collision probability calculation unit generates N=1000 groups of trajectories by Monte Carlo simulation, and the collision probability P calculation formula is:

[0046]

[0047] wherein x car (t) is the vehicle position, x src (t) is the sound source position, ||·|| represents the distance between two points, and the collision probability reflects the proximity of the sound source and the vehicle;

[0048] The dynamic decision unit triggers an emergency warning if P>70% and the remaining time is less than 2s.

[0049] As a preferred scheme of the present application, the warning module comprises:

[0050] The multi-state prompting unit displays the sound source position on the vehicle screen in a 30° sector area, and simultaneously issues a voice prompt;

[0051] The emergency braking interface issues a deceleration instruction if the collision probability P>90% and the driver response time is greater than 1s.

[0052] The log recording unit saves data for 30 days.

[0053] As a preferred scheme of the present application, the self-learning module comprises:

[0054] The federated learning unit updates the model through the federated learning framework, and optimizes the objective function:

[0055]

[0056] where: θ is a parameter of the model, which needs to be optimized by training; denotes the total size of all data sets; L denotes a loss function; f(x (i) ; θ) is the output of the i-th client under the global model parameter θ; y (i) is the corresponding label; D i is the sample set of the i-th client;

[0057] A scene self-adapting unit adjusts the early warning sensitivity according to the area type.

[0058] A model calibration unit optimizes the noise compensation coefficient every 24 hours.

[0059] A vehicle-mounted sound recognition and positioning intelligent early warning method, comprising the following steps:

[0060] Step 1: Real-time acquisition of audio signals of the surrounding environment through multiple sound detectors, and spatio-temporal calibration of the signals of multiple detectors to synchronize the data of different devices;

[0061] Step 2: Analysis of the acquired audio signals, identification of the sound source type, and judgment of whether the sound source is blocked in front by combining the vehicle-mounted camera and radar data, wherein the sound source type includes pedestrians, tricycles, motorcycles and other motor vehicles;

[0062] Step 3: Calculation of the azimuth of the sound source according to the time difference of arrival TDOA of the signals of multiple sound detectors, and calculation of the distance of the sound source combined with the sound intensity attenuation model, and optimization of the distance estimation result by using the particle swarm optimization PSO algorithm;

[0063] Step 4: Fusion of the signal data of the vehicle-mounted camera and radar through extended Kalman filtering EKF, correction of the sound source position and optimization of the positioning result, and dynamic adjustment of the noise compensation coefficient according to the environmental noise intensity;

[0064] Step 5: Evaluation of the collision risk based on the spatial position, blocking condition, object type and distance of the sound source, and generation of early warning signals of different levels, and triggering of emergency early warning when the collision risk exceeds the set threshold and the remaining time is less than 2 seconds;

[0065] Step 6: Issuance of early warning information to the driver through the vehicle-mounted display screen or voice prompt, and triggering of the automatic deceleration instruction if the collision risk meets the preset condition;

[0066] Step 7: Updating of the model through the federated learning framework, and dynamic adjustment of the early warning sensitivity and noise compensation coefficient according to the area type.

[0067] As a preferred scheme of the present application, the sound source identification step comprises:

[0068] The audio signals collected by multiple sound detectors are denoised and preprocessed, and the mel frequency cepstral coefficient (MFCC) and linear predictive coding (LPC) features of each signal are extracted;

[0069] The extracted audio features are fused, and the MFCC and LPC feature vectors are combined through a weighted splicing method to improve the expression ability of the audio features;

[0070] The fused feature vectors are classified using a deep residual network (ResNet), the sound source type is identified, and the sound source type label is output, including pedestrians, tricycles, motorcycles and other motor vehicles;

[0071] The visual signals of the vehicle-mounted camera and radar are combined to determine whether the sound source object is blocked in front, and if the target cannot be detected in the predicted area, it is determined to be blocked.

[0072] As a preferred scheme of the present application, the positioning step comprises:

[0073] The time difference of arrival (TDOA) of multiple sound detector signals is calculated, and the generalized cross-correlation phase transform (GCC-PHAT) algorithm is used to obtain the azimuth information of the sound source;

[0074] According to the sound intensity attenuation model, the distance of the sound source is calculated, and the particle swarm optimization (PSO) algorithm is used to optimize the distance estimation result;

[0075] The signal data of the vehicle-mounted camera and radar are fused through the extended Kalman filter (EKF) algorithm, the sound source position is corrected, and the positioning result is optimized;

[0076] According to the real-time environmental noise intensity, the noise compensation coefficient is adjusted to improve the positioning accuracy.

[0077] Compared with the prior art, the beneficial effects of the present application are: the present application significantly improves the performance of the vehicle intelligent early warning system by fusing the data of multiple sensors, especially combining sound detection technology with vehicle-mounted cameras and radar data. The system can collect and analyze audio signals of the surrounding environment in real time, and combine visual and radar data of other sensors to effectively overcome the collision detection blind area caused by limited or blocked sensor view in traditional technology. Through deep learning model analysis of audio signals, the system can accurately identify and judge the sound source type in complex environment, improving the recognition accuracy and reliability of the system. At the same time, through the precise space-time calibration of the calibration module, the system can accurately align the audio signals with the vehicle-mounted camera and radar detection data, ensuring the accurate fusion of multi-sensor data, solving the space-time synchronization problem between sensor data, and improving the accuracy of sound source position calculation. The positioning module calculates the direction of the sound source through the time difference TDOA of multiple sound detectors signals, and estimates the distance by combining the sound intensity attenuation model and particle swarm optimization algorithm, and the extended Kalman filter further corrects the positioning result, ensuring the accuracy of sound source positioning. The system dynamically adjusts the noise compensation coefficient according to the environmental noise intensity, optimizes the noise compensation in real time, reduces the interference of environmental noise on the system recognition and positioning results, and enhances the stability of the system in noisy environment. The risk assessment module generates different levels of warning signals by evaluating the spatial position, shielding state, object type and distance of the sound source, triggers automatic deceleration instructions in time, and reduces the risk of collision. Through the federated learning framework, the self-learning module can continuously update the model, dynamically adjust the warning sensitivity and noise compensation coefficient, and ensure that the system can adapt to different driving environments and continuously improve its performance. BRIEF DESCRIPTION OF DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0079] Among them:

[0080] Figure 1 The system modular structure diagram of the present application;

[0081] Figure 2 The method flowchart of the present application. DETAILED DESCRIPTION

[0082] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0083] As Figure 1 shown, one embodiment of the present application provides a vehicle-mounted sound recognition and positioning intelligent early warning system, comprising:

[0084] (1) Sound detector module

[0085] for real-time acquisition of audio signals of the surrounding environment;

[0086] In this embodiment, the module includes a plurality of sound detectors installed at different positions of the vehicle-mounted system for real-time acquisition of audio signals in the surrounding environment. The sound detector can perceive various sound signals from outside the vehicle, including the sound of the engine of the vehicle during driving, the sound of pedestrians, tricycles, motorcycles and other road users. Through a high-sensitivity microphone array, the sound detector can still effectively capture the surrounding sound signals under the condition of fast vehicle speed or noisy external environment, and transmit them to the subsequent sound recognition module.

[0087] (2) Sound recognition module

[0088] for analyzing the collected audio signals through a deep learning model, identifying the type of sound source, and determining whether the sound source is blocked in combination with the vehicle-mounted camera and radar data;

[0089] The sound recognition module comprises:

[0090] a timbre feature extraction unit extracts a feature vector of the audio signal through a mel-frequency cepstral coefficient (MFCC) and linear predictive coding (LPC) fusion algorithm. The fusion algorithm combines the feature vectors of MFCC and LPC through a weighted splicing method. The specific steps are as follows:

[0091] a. Preprocessing of the audio signal, including denoising and framing operation;

[0092] b. Using the mel-frequency cepstral coefficient (MFCC) algorithm to extract the spectral features of the audio signal to obtain a feature vector:

[0093] MFCC = [mfcc1, mfcc2,..., mfcc N ];

[0094] Where: mfcc NNth eigenvalue of the mel-frequency cepstral coefficient; N is the dimension of the MFCC feature;

[0095] c. using the linear predictive coding (LPC) algorithm to extract the prediction coefficient feature of the audio signal, and obtaining a feature vector:

[0096] LPC = [lpc1, lpc2, …, lpc M ];

[0097] wherein: lpc M Mth eigenvalue of the LPC feature; M is the dimension of the LPC feature;

[0098] d. fusing the MFCC and LPC feature vectors by weighted splicing to obtain a fused feature vector:

[0099] F fusion = [mfcc1, mfcc2, …, mfcc N , lpc1, lpc2, …, lpc M ];

[0100] wherein: F fusion is the final fused feature vector, N and M are the dimensions of the MFCC and LPC features respectively, and in the weighted splicing, the weight coefficients of the MFCC and LPC are adjusted according to the application environment or training data;

[0101] a classification unit for classifying the fused audio feature vector based on a deep residual network (ResNet) model and outputting a sound source type label, including pedestrians, tricycles, motorcycles and other motor vehicles, and the classification method is suitable for variable and complex noise environments through a deep learning model;

[0102] an occlusion judgment unit for determining that the sound source object is occluded when the vehicle-mounted camera or radar fails to detect the target in the predicted sound source area, and the occlusion judgment method improves the traditional occlusion detection based on visual sensors and combines the fusion of sound signals and visual signals.

[0103] In this embodiment, the sound recognition module is used to analyze the collected audio signals and identify the type of sound source. This module uses deep learning models (such as convolutional neural networks CNN, recurrent neural networks RNN, etc.) for feature extraction and classification of audio signals. First, the audio signal extracts the audio features through the Mel-frequency cepstral coefficient (MFCC) and linear predictive coding (LPC), and then inputs the features into the deep learning model for sound source classification. The sound source types include pedestrians, tricycles, motorcycles, and other motor vehicles. The sound recognition module also combines the vehicle-mounted camera and radar data to determine whether the sound source is blocked through multi-modal fusion of image and sound data. For example, in the case where the vehicle-mounted camera cannot clearly identify, the system can rely on the sound source positioning information provided by the sound recognition module to supplement the lack of visual data.

[0104] The core idea of the weighted splicing method is that when splicing the MFCC and LPC feature vectors, considering that the importance of these two features may be different in different driving environments, we introduce dynamic weighting coefficients to optimize the combination of these two features. The MFCC feature has good performance for the spectral characteristics of the sound, while the LPC feature can capture more detailed changes in the speech signal. According to the noise intensity, vehicle speed and external interference of different environments, the weight coefficient of each feature is dynamically adjusted, thereby improving the accuracy and accuracy of sound source recognition.

[0105] MFCC: Mel-frequency cepstral coefficient, simulating human auditory characteristics, extracting spectral features through steps such as framing, windowing, FFT, Mel filter bank, discrete cosine transform, etc.

[0106] LPC: Linear predictive coding, predicting the current value by linearly combining past samples to extract channel characteristics parameters.

[0107] (3) Calibration module

[0108] Used for precise space-time calibration of the audio signal with vehicle-mounted camera and radar detection data, optimizing signal alignment through time synchronization and space mapping to determine whether the sound source object is blocked;

[0109] The calibration module includes:

[0110] Time synchronization unit: uses precise time protocol PTP (PTP is a high-precision time synchronization protocol used to calibrate sensor clocks) to perform time calibration on each sensor, with a synchronization error Δt ≤ 1ms;

[0111] Space mapping unit: maps the angle of the sound source position to the angle of the vehicle-mounted camera field of view, with the mapping formula as:

[0112]

[0113] wherein: θ is the horizontal angle of the sound source relative to the vehicle-mounted camera, φ is the vertical angle of the sound source relative to the vehicle-mounted camera, X source and Y source are the lateral and longitudinal coordinates of the sound source; D is the distance between the sound source and the vehicle-mounted camera;

[0114] The occlusion detection unit marks the source as an occluded target if the vehicle-mounted camera or radar fails to detect a target within the sound source prediction area, and the sound source category is a person or other motor vehicle.

[0115] In this embodiment, the calibration module is used to precisely calibrate the collected audio signals with the vehicle-mounted camera and radar detection data. This module uses time synchronization and spatial mapping technology to align data from different sensors, ensuring that the data from each sensor can be processed in the same spatiotemporal framework. Time synchronization is achieved through the Precision Time Protocol (PTP), while the spatial mapping algorithm corresponds the angle of the sound source position to the field of view angle of the vehicle-mounted camera. The calibration module can accurately determine whether the sound source object is occluded, and provide accurate spatial coordinate data for subsequent sound source positioning.

[0116] (4) Positioning module

[0117] The positioning module is used to calculate the azimuth of the sound source object through the time difference of arrival TDOA of multiple sound detector signals, calculate the distance of the sound source object in combination with the sound intensity decay model and particle swarm optimization PSO algorithm, and correct the positioning result by fusing camera and radar data through extended Kalman filtering EKF, while dynamically adjusting the noise compensation coefficient according to the environmental noise intensity.

[0118] The positioning module includes:

[0119] The azimuth calculation unit uses the generalized cross-correlation phase transform GCC-PHAT algorithm to calculate the time delay difference between the sound detectors.

[0120] The distance estimation unit uses the particle swarm optimization PSO algorithm to fit and calculate the distance of the sound source based on the sound intensity decay model I(d) = I0-20log 10 (d)-αd+ò, where I(d) represents the signal intensity at a distance d; I0represents the signal intensity at a reference distance; d represents the distance between the sound source and the detector; α represents the sound intensity decay coefficient, which is usually related to the propagation environment and frequency; and ò represents the noise term, which is a random error.

[0121] The position correction unit optimizes the position calculation by fusing multi-source data through extended Kalman filtering EKF when the camera / radar detects that the target is not occluded.

[0122] An environmental noise compensation unit monitors environmental noise intensity in real time, dynamically adjusts the noise source term, and the noise model is σ ~ N(0, σ 2 ), where σ represents the noise standard deviation; N(0, σ 2 ) represents a Gaussian distribution with a mean of 0 and a variance of σ 2 , representing environmental noise.

[0123] In this embodiment, the function of the positioning module is to calculate the direction of the sound source object through the time difference of arrival (TDOA) of the signals of multiple sound detectors. The module uses a sound intensity attenuation model combined with a particle swarm optimization (PSO) algorithm to calculate the distance of the sound source. The sound intensity attenuation model is used to describe the law of sound attenuation with increasing distance, and the PSO algorithm optimizes the distance estimation result by simulating particle swarm search, thereby achieving more accurate distance measurement. In addition, the positioning module fuses vehicle-mounted camera and radar data through an extended Kalman filter (EKF) algorithm to further correct the position of the sound source and improve the positioning accuracy. According to the environmental noise intensity, the system can dynamically adjust the noise compensation coefficient, optimize the processing effect of the sound signal, and reduce the influence of environmental noise on the positioning accuracy.

[0124] (5) Risk assessment module

[0125] for assessing collision risk based on the spatial position, occlusion state, object type and distance of the sound source, and generating different levels of warning signals;

[0126] The risk assessment module comprises:

[0127] a risk classification unit: set three levels of warning thresholds, classify according to distance d: near d≤5m; medium 5

[0128]

[0129] where: d0=10m, k=0.5m -1 , indicating the relationship between distance and alarm intensity;

[0130] a trajectory prediction unit based on long short-term memory network LSTM to predict the future T=2s sound source position and predict the future motion trajectory of the sound source;

[0131] a collision probability calculation unit that generates N=1000 sets of trajectories using Monte Carlo simulation, and the collision probability P is calculated by the formula:

[0132]

[0133] where: x car (t) is the vehicle position, x src(t) is the sound source position, ||·|| represents the distance between two points, and the collision probability reflects the proximity of the sound source to the vehicle;

[0134] A dynamic decision unit triggers an emergency warning if P > 70% and the remaining time is less than 2s.

[0135] In this embodiment, the risk assessment module is used to assess potential collision risks and generate different levels of warning signals according to the risk level. This module considers factors such as the spatial position of the sound source, the occlusion state, the object type, and the relative distance to the vehicle. The risk assessment module determines the severity of the collision risk by setting different warning thresholds and triggers the corresponding warning response according to the evaluation results. When the collision risk exceeds the set threshold, the system will issue a high-risk warning signal. The role of this module is to ensure that the driver can timely understand the potential danger and take appropriate avoidance measures.

[0136] (6) Warning module

[0137] It is used to display the position of the sound source on the vehicle display screen and issue a voice prompt, and trigger an automatic deceleration instruction when the collision risk meets the preset conditions;

[0138] The warning module includes:

[0139] A multi-state prompt unit displays the position of the sound source on the vehicle screen in 30° sector areas and issues a voice prompt;

[0140] Emergency braking interface: if the collision probability P > 90% and the driver's response time is greater than 1s, issue a deceleration instruction;

[0141] Log recording unit, data retention period is 30 days.

[0142] In this embodiment, the warning module is responsible for displaying the position of the sound source on the vehicle display screen and prompting the driver through voice. When the collision risk meets the set conditions, the system will trigger an automatic deceleration instruction to help the vehicle slow down and avoid danger. The warning module displays the position information of the sound source in real time, helping the driver understand the potential obstacles in the surrounding environment, and providing sound prompts or deceleration reminders at critical moments, thereby reducing the probability of accidents.

[0143] (7) Self-learning module

[0144] It is used to update the model through the federated learning framework and dynamically adjust the warning sensitivity and noise compensation coefficient according to the region type.

[0145] The self-learning module includes:

[0146] A federated learning unit updates the model through the federated learning framework and optimizes the objective function:

[0147]

[0148] where: θ is a parameter of the model, which needs to be optimized through training; denotes the total size of all data sets; L denotes a loss function; f(x (i) ; θ) is the output of the i-th client under the global model parameter θ; y (i) is the corresponding label; D i is the sample set of the i-th client;

[0149] a scene self-adapting unit, which adjusts the early warning sensitivity according to the area type;

[0150] a model calibration unit, which optimizes the noise compensation coefficient every 24 hours.

[0151] In this embodiment, the self-learning module continuously updates and optimizes the system model through the federated learning framework. This module enables the system to dynamically adjust the early warning sensitivity and noise compensation coefficient according to the area type, thereby realizing self-adaptive learning. The federated learning framework enables different vehicles to share experience while ensuring data privacy, thereby continuously improving system performance. The addition of the self-learning module enables the system to continuously optimize according to different environments and driving scenarios, thereby improving its performance in various complex driving conditions.

[0152] Federated learning allows multiple clients (vehicles) to collaboratively train a shared model without sharing raw data. In the self-learning module, each vehicle updates the model parameters locally and uploads them to the server for aggregation, thereby improving the performance of the global model.

[0153] As shown in FIG. 1, another embodiment of the present application provides a vehicle-mounted sound recognition and positioning intelligent early warning method, which comprises the following steps: Figure 2

[0154] Step 1: Real-time acquisition of audio signals of the surrounding environment through multiple sound detectors, and spatio-temporal calibration of the signals of multiple detectors to synchronize the data of different devices;

[0155] Step 2: Analysis of the acquired audio signals, identification of the sound source type, and judgment of whether the sound source is blocked in front by combining the vehicle-mounted camera and radar data, wherein the sound source type includes pedestrians, tricycles, motorcycles and other motor vehicles;

[0156] Step 3: Calculation of the direction of the sound source according to the time difference of arrival TDOA of the signals of multiple sound detectors, and calculation of the distance of the sound source by combining the sound intensity attenuation model, and optimization of the distance estimation result by using the particle swarm optimization PSO algorithm;

[0157] ​Step 4: Fuse the signal data of the vehicle-mounted camera and radar through the Extended Kalman Filter (EKF) to correct the sound source position and optimize the positioning result, and dynamically adjust the noise compensation coefficient according to the environmental noise intensity;

[0158] Step 5: Based on the spatial position of the sound source, the occlusion situation, the object type and the distance, evaluate the collision risk and generate different levels of warning signals, and trigger an emergency warning when the collision risk exceeds the set threshold and the remaining time is less than 2 seconds;

[0159] Step 6: Send warning information to the driver through the vehicle-mounted display screen or voice prompt, and trigger the automatic deceleration instruction if the collision risk meets the preset conditions;

[0160] Step 7: Update the model through the federated learning framework and dynamically adjust the warning sensitivity and noise compensation coefficient according to the region type.

[0161] Specifically, the sound source identification step includes:

[0162] Denoise and preprocess the audio signals collected by multiple sound detectors, extract the Mel Frequency Cepstral Coefficient (MFCC) and Linear Predictive Coding (LPC) features of each signal;

[0163] Fuse the extracted audio features, combine the MFCC and LPC feature vectors through weighted splicing to improve the expression ability of the audio features;

[0164] Use a deep residual network (ResNet) to classify the fused feature vectors, identify the sound source type, and output the sound source type label, including pedestrians, three-wheeled vehicles, motorcycles and other motor vehicles;

[0165] Combine the visual signals of the vehicle-mounted camera and radar to determine whether the sound source object is occluded in front, and if the target cannot be detected within the predicted area, it is determined to be occluded.

[0166] The positioning step includes:

[0167] Calculate the Time Difference of Arrival (TDOA) of multiple sound detector signals, and use the Generalized Cross Correlation Phase Transform (GCC-PHAT) algorithm to obtain the azimuth information of the sound source;

[0168] According to the sound intensity attenuation model, calculate the distance of the sound source, and optimize the distance estimation result through the Particle Swarm Optimization (PSO) algorithm;

[0169] Fuse the signal data of the vehicle-mounted camera and radar through the Extended Kalman Filter (EKF) algorithm to correct the sound source position and optimize the positioning result;

[0170] Adjust the noise compensation coefficient according to the real-time environmental noise intensity to improve the positioning accuracy.

[0171] In summary, the present application proposes a brand-new vehicle-mounted sound recognition and positioning intelligent early warning system by innovatively integrating sound detection technology with vehicle-mounted cameras and radar data. This system can effectively solve the collision detection blind area problem caused by limited sensor visibility, obstruction or environmental noise interference in the prior art, significantly improving the response speed and accuracy of the vehicle-mounted intelligent early warning system in complex environments. Through the fusion of multi-sensor data and the introduction of deep learning models, the system can analyze and locate potential obstacles in real time, assess collision risks, and provide timely and effective warnings. In particular, through the self-learning module and dynamic adjustment mechanism, the system can continuously optimize the warning sensitivity and noise compensation coefficient, ensuring efficient operation in various complex scenarios. This technical solution not only has strong environmental adaptability, but also significantly improves the safety of drivers, and has broad application prospects and commercial value.

[0172] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0173] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions or other processes. And the scope of preferred embodiments of the present application includes additional implementation in which the functions are performed in different orders, in substantially simultaneous fashion, or in reverse order, depending on the functionality involved.

[0174] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, and these should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An intelligent early warning system for vehicle sound recognition and positioning, characterized in that, The application relates to a sound source detection system for intelligent vehicles, which comprises the following modules: a sound detector module for collecting audio signals of the surrounding environment in real time; a sound recognition module for analyzing the collected audio signals through a deep learning model, identifying the sound source type, and determining whether the sound source is blocked by combining vehicle-mounted camera and radar data; a calibration module for precisely calibrating the audio signals with vehicle-mounted camera and radar detection data, optimizing signal alignment through time synchronization and space mapping, and determining whether the sound source object is blocked; a positioning module for calculating the direction of the sound source object through the time difference of arrival (TDOA) of multiple sound detector signals, calculating the distance of the sound source object by combining the sound intensity attenuation model and the particle swarm optimization (PSO) algorithm, and correcting the positioning result by fusing camera and radar data through extended Kalman filtering (EKF) while dynamically adjusting the noise compensation coefficient according to the environmental noise intensity; a risk assessment module for assessing the collision risk based on the spatial position, blocking state, object type, and distance of the sound source, and generating different levels of warning signals; a warning module for displaying the sound source position on the vehicle-mounted display screen and issuing a voice prompt, and triggering an automatic deceleration instruction when the collision risk meets the preset conditions; a self-learning module for updating the model through a federated learning framework and dynamically adjusting the warning sensitivity and noise compensation coefficient according to the region type.

2. The vehicle-mounted sound recognition and positioning intelligent early warning system according to claim 1, characterized in that, The sound recognition module comprises: a timbre feature extraction unit that extracts the feature vector of the audio signal through a mel-frequency cepstral coefficient (MFCC) and linear predictive coding (LPC) fusion algorithm. The fusion algorithm combines the feature vectors of MFCC and LPC through a weighted splicing method. The specific steps are as follows: a. Preprocessing the audio signal, including denoising and framing operation; b. Extracting the spectral features of the audio signal using the mel-frequency cepstral coefficient (MFCC) algorithm to obtain the feature vector: MFCC = [mfcc1, mfcc2,..., mfcc N ]; wherein: mfcc N denotes the Nth eigenvalue of the mel-frequency cepstral coefficient; N is the dimension of the MFCC feature; c. Extracting the prediction coefficient features of the audio signal using the linear predictive coding (LPC) algorithm to obtain the feature vector: LPC = [lpc1, lpc2,..., lpc M ]; wherein: lpc M denotes the Mth eigenvalue of the LPC feature; M is the dimension of the LPC feature; d. Fusing the MFCC and LPC feature vectors through a weighted splicing method to obtain the fused feature vector: F fusion = [mfcc1, mfcc2,..., mfcc N , lpc1, lpc2,..., lpc M ]; where F fusion is the final fusion feature vector, N and M are the dimensions of MFCC and LPC features respectively, and the weight coefficients of MFCC and LPC are adjusted according to the application environment or training data in the weighted concatenation. a classification unit for classifying the fused audio feature vector based on a deep residual network (ResNet) model and outputting the sound source type label, including pedestrians, three-wheeled vehicles, motorcycles, and other motor vehicles. The classification method is suitable for variable and complex noise environments through a deep learning model; an occlusion judgment unit for determining that the sound source object is blocked when the vehicle-mounted camera or radar fails to detect the target in the predicted sound source area. The occlusion judgment method improves the traditional occlusion detection based on visual sensors by combining sound signals and visual signals.

3. The vehicle-mounted sound recognition and positioning intelligent early warning system according to claim 1, characterized in that, The calibration module comprises: a time synchronization unit that uses the precision time protocol (PTP) to calibrate the time of each sensor, with a synchronization error Δt ≤ 1 ms; a space mapping unit that maps the angle of the sound source position with the angle of the vehicle-mounted camera field of view, with the mapping formula being: where: θ is the horizontal angle of the sound source relative to the vehicle camera, φ is the vertical angle of the sound source relative to the vehicle camera, X source and Y source are the lateral and longitudinal coordinates of the sound source; and D is the distance of the sound source from the vehicle camera. an occlusion detection unit that marks the source as a blocked target if the vehicle-mounted camera or radar fails to detect the target in the predicted sound source area and the sound source category is a person or other motor vehicle.

4. The vehicle-mounted sound recognition and positioning intelligent early warning system according to claim 1, characterized in that, The positioning module comprises: An azimuth calculation unit calculates the time delay difference between the sound detectors using the generalized cross-correlation phase transform (GCC-PHAT) algorithm; a distance estimation unit, based on the sound intensity decay model I(d) = I0-20log 10 (d)-ad+0, using a particle swarm optimization (PSO) algorithm to fit the sound source distance; wherein: I(d) represents the signal intensity at a distance d; I0represents the signal intensity at a reference distance; d represents the distance between the sound source and the detector; a represents the sound intensity decay coefficient, which is usually related to the propagation environment and frequency; 0 represents a noise term, which is a random error; A position correction unit optimizes the position calculation by fusing multi-source data through extended Kalman filtering (EKF) when the camera / radar detects that the target is not blocked; The environmental noise compensation unit monitors the environmental noise intensity in real time, dynamically adjusts the noise source term, and the noise model is σ ~ N(0, σ 2 ), wherein: σ represents the standard deviation of noise; N(0, σ 2 ) represents a Gaussian distribution with a mean of 0 and a variance of σ 2 , representing environmental noise.

5. The vehicle-mounted sound recognition and positioning intelligent early warning system according to claim 1, characterized in that, The risk assessment module includes: A risk classification unit sets three early warning threshold values and classifies them according to the distance d: near d≤5m; medium 5 where: d0= 10 m, k = 0.5 m -1 , represents the relationship between distance and alarm intensity; A trajectory prediction unit predicts the future T=2s sound source position based on the long short-term memory (LSTM) network, and predicts the future motion trajectory of the sound source; A collision probability calculation unit generates N=1000 trajectories using Monte Carlo simulation, and the collision probability P is calculated using the following formula: where: x car (t) is the vehicle position, x src (t) is the sound source position, ||·|| represents the distance between two points, and the collision probability reflects the proximity of the sound source to the vehicle; A dynamic decision unit triggers an emergency warning if P>70% and the remaining time is less than 2s.

6. The vehicle-mounted sound recognition and positioning intelligent early warning system according to claim 1, characterized in that, The warning module includes: A multi-state prompt unit displays the sound source position in 30° sectors on the vehicle screen and issues a voice prompt; An emergency braking interface issues a deceleration command if the collision probability P>90% and the driver's response time is greater than 1s; A log recording unit saves data for 30 days.

7. The vehicle-mounted sound recognition and positioning intelligent early warning system according to claim 1, characterized in that, The self-learning module includes: A federated learning unit updates the model through a federated learning framework to optimize the objective function: where: θ is a parameter of the model, which needs to be optimized through training; represents the total size of all data sets; L represents a loss function; f(x (i) ; θ) is the output of the i-th client under the global model parameter θ; y (i) is the corresponding label; D i is the sample set of the i-th client; A scene adaptive unit adjusts the warning sensitivity according to the area type; A model calibration unit optimizes the noise compensation coefficient every 24 hours. 8.The in-vehicle sound recognition and positioning intelligent warning method according to any one of claims 1-7, characterized in that, The method includes the following steps: Step 1: Real-time acquisition of audio signals of the surrounding environment through multiple sound detectors, and spatio-temporal calibration of signals from multiple detectors to synchronize data from different devices; Step 2: Analysis of the collected audio signals to identify the sound source type, and judgment of whether the sound source is blocked by the front based on vehicle-mounted camera and radar data, the sound source type including pedestrians, tricycles, motorcycles, and other motor vehicles; Step 3: Calculation of the azimuth of the sound source based on the time difference of arrival (TDOA) of signals from multiple sound detectors, and calculation of the distance of the sound source based on the sound intensity attenuation model, with the particle swarm optimization (PSO) algorithm used to optimize the distance estimation results; Step 4: Fusion of vehicle-mounted camera and radar signal data through extended Kalman filtering (EKF) to correct the sound source position and optimize the positioning results, and dynamic adjustment of the noise compensation coefficient according to the environmental noise intensity; Step 5: Evaluation of the collision risk based on the spatial position of the sound source, the blocking condition, the object type, and the distance, and generation of different levels of warning signals, with an emergency warning triggered when the collision risk exceeds the set threshold and the remaining time is less than 2 seconds; Step 6: Issuance of warning information to the driver through the vehicle-mounted display screen or voice prompt, with an automatic deceleration command triggered if the collision risk meets the preset conditions; Step 7: Updating of the model through a federated learning framework, and dynamic adjustment of the warning sensitivity and noise compensation coefficient according to the area type.

9. The vehicle-mounted sound recognition and positioning intelligent early warning method according to claim 8, characterized in that, The sound source identification step includes: Denoising and preprocessing of the audio signals collected by multiple sound detectors to extract the mel frequency cepstral coefficient (MFCC) and linear predictive coding (LPC) features of each signal; The extracted audio features are fused, and MFCC and LPC feature vectors are combined through weighted splicing to improve the expression ability of the audio features; A deep residual network ResNet is used to classify the fused feature vectors, identify the sound source type, and output the sound source type label, including pedestrians, tricycles, motorcycles and other motor vehicles; The visual signals of the vehicle-mounted camera and radar are combined to determine whether the sound source object is blocked in front, and if the target cannot be detected in the predicted area, it is determined to be blocked.

10. The vehicle-mounted sound recognition and positioning intelligent early warning method according to claim 8, characterized in that, The positioning step comprises: The time difference of arrival TDOA of a plurality of sound detector signals is calculated, and the generalized cross-correlation phase transform GCC-PHAT algorithm is used to obtain the azimuth information of the sound source; According to the sound intensity attenuation model, the distance of the sound source is calculated, and the distance estimation result is optimized through the particle swarm optimization PSO algorithm; the signal data of the vehicle-mounted camera and radar are fused through the extended Kalman filter EKF algorithm, the sound source position is corrected, and the positioning result is optimized; According to the real-time environmental noise intensity, the noise compensation coefficient is adjusted to improve the positioning accuracy.

Images