Acoustic non-line-of-sight vehicle approach and departure detection method

Abstract

The application provides an acoustic non-line-of-sight vehicle approaching and moving away detection method, comprising: constructing a vehicle approaching and moving away detection model, a feature extraction module receiving multi-channel audio at a vehicle intersection and outputting a spectrum graph of first channel audio data and a continuous sound energy distribution conversion image of multi-channel audio data, a double-path convolution-long short-term memory module receiving the spectrum graph and the image and outputting audio data time features and spatial features, and a detection output module receiving the features and outputting a vehicle approaching and moving away detection result; collecting complete multi-channel audio and video data and training the model; obtaining multi-channel audio data of the surrounding environment at the intersection and inputting the trained detection model to obtain a detection result. The application does not depend on strong hypotheses, introduces a spectrum graph as a feature representation, extracts time features of a deep representation through a long short-term memory network module, can combine an acoustic non-line-of-sight vehicle detection task and a positioning task, and improves the classification precision of the non-line-of-sight vehicle approaching and moving away state.

Description

Technical Field

[0001] This invention belongs to the field of acoustic scene perception technology, specifically relating to an acoustic non-line-of-sight vehicle approach and departure detection method. Background Technology

[0002] Current vehicle sensing systems primarily rely on objects within direct line-of-sight. When the line of sight is obstructed by obstacles such as walls, these systems fail to detect non-line-of-sight vehicles, posing a risk of collision. Therefore, research into non-line-of-sight vehicle perception methods is crucial for traffic safety. Even with obstacles between the sound source and the microphone, the sound emitted by the source can still reach the microphone after various wave interactions. By using acoustic sensors to acquire ambient sound and employing intelligent sensing technology to extract the location and proximity / distance status of non-line-of-sight vehicles in the acoustic environment, acoustic traffic perception capabilities can be provided for applications such as traffic monitoring and autonomous driving, compensating for the adverse effects of line-of-sight obstruction on traffic scene perception.

[0003] By utilizing onboard passive acoustic sensing systems, non-line-of-sight vehicle detection can be achieved by monitoring ambient sound. Acoustic non-line-of-sight vehicle detection and localization methods can be broadly classified into two categories: computational acoustics-based methods and data-driven methods.

[0004] In computational acoustics, geometric acoustics ignores the wave properties of sound and studies sound propagation using geometric methods, while wave acoustics mainly considers the diffraction effect of sound waves.

[0005] In the data-driven approach, the acoustic energy spectrum generated during the sound source localization process is used as input, and an SVM classifier is used to model the correspondence between the acoustic energy spectrum and the real location of the non-line-of-sight vehicle in the current environment. Summary of the Invention

[0006] The inventors of this application have discovered that the above method has at least the following technical problems:

[0007] To simulate the impact of occlusion on acoustic sensors, computational acoustics methods rely on strong assumptions, such as the existence of only a single propagation mode like reflection or diffraction for non-line-of-sight sound source localization, without considering the influence of multiple acoustic propagation modes. Moreover, non-line-of-sight sound source localization methods based on reflection are only applicable to high-frequency sound sources, and non-line-of-sight sound source localization methods based on diffraction are only applicable to low-frequency sound sources. However, the sound emitted by a vehicle in motion covers both low and high frequencies. Therefore, computational acoustics methods based on a single acoustic propagation mode cannot accurately locate non-line-of-sight vehicles.

[0008] Furthermore, acoustic non-line-of-sight (NLOS) vehicle detection and localization are two distinct branches. Existing computational acoustic methods and data-driven acoustic NLOS vehicle proximity detection methods only utilize phase information from multi-channel audio, lacking feature representations for acoustic NLOS vehicle detection, and therefore cannot determine whether the sound originates from a vehicle. Jointly performing acoustic NLOS vehicle detection and localization tasks can effectively avoid perceptual errors; however, current technologies lack research on joint models for these two tasks.

[0009] The approach and departure of non-line-of-sight vehicles can occur over a period of time. Classification methods that preserve temporal context information along feature representations can extract temporal features, which is beneficial for improving detection accuracy. However, existing acoustic non-line-of-sight vehicle approach detection methods do not use SVM classifiers to extract temporal features, thus preventing the use of temporal features for detection.

[0010] The purpose of this invention is to address the shortcomings of existing acoustic non-line-of-sight (NLOS) vehicle detection methods, which rely on strong assumptions, leading to inaccurate NLOS vehicle localization, and lack of acoustic NLOS vehicle detection feature representation, thus preventing the joint acoustic NLOS vehicle detection and localization tasks. This invention provides an acoustic NLOS vehicle approach and distance detection method that does not rely on strong assumptions. It introduces a spectrogram as a feature representation and extracts temporal features of the depth representation through a long short-term memory (LSTM) network module. This method can jointly perform acoustic NLOS vehicle detection and localization tasks, improving the accuracy of NLOS vehicle detection and the classification of NLOS vehicle approach and distance states.

[0011] To achieve the above objectives, the technical solution provided by this invention is:

[0012] An acoustic non-line-of-sight vehicle approach and departure detection method, characterized by the following steps:

[0013] Step 1: Construct an acoustic non-line-of-sight vehicle approach and departure detection model. The detection model includes a feature extraction module, a dual-path convolutional-long short-term memory module, and a detection output module.

[0014] The feature extraction module receives multi-channel audio from vehicles passing through the intersection and outputs a spectrogram of the first channel audio data and a continuous sound energy distribution conversion image of the multi-channel audio data.

[0015] The dual-channel convolutional-long short-term memory module receives the spectrogram and continuous acoustic energy distribution transformation image extracted by the feature extraction module, and outputs the temporal and spatial features of the multi-channel audio data.

[0016] The detection output module is used to receive the temporal and spatial features extracted by the dual-channel convolution-long short-term memory module, and output the non-line-of-sight vehicle approach and departure detection results with directional information;

[0017] Step 2: Collect complete multi-channel audio and video data of vehicles passing through the intersection and train the constructed detection model.

[0018] Step 3: Acquire multi-channel audio data of the environment surrounding the intelligent vehicle at the intersection, and input the acquired multi-channel audio data into the trained detection model to obtain the detection results of non-line-of-sight vehicles approaching and moving away.

[0019] Furthermore, the feature extraction module calculates the sound energy distribution of the continuous surrounding environment from the received multi-channel audio, and then uses max-min normalization to transform the continuous sound energy distribution into a two-dimensional image representation.

[0020] Furthermore, the feature extraction module uses SRP-PHAT to calculate the sound energy distribution of the continuous surrounding environment from the received multi-channel audio.

[0021] Furthermore, the feature extraction module uses the STFT method to generate a spectrogram by applying it to the first channel audio of the received multi-channel audio.

[0022] Furthermore, step 2 includes the following sub-steps:

[0023] Step 2.1: Collect complete multi-channel audio and video data of the vehicle passing through the intersection;

[0024] Step 2.2: Divide the collected multi-channel audio segments containing the complete process of a vehicle passing through an intersection into a training set and a validation set;

[0025] Step 2.3: Based on the times when the vehicle appears and leaves the field of vision recorded in the video, the entire segment of multi-channel audio data of the vehicle passing through the intersection is segmented and classified. Multi-channel audio samples are extracted from the segmented audio segments through a sliding time window.

[0026] Step 2.4: Train the constructed detection model using the divided training and validation sets.

[0027] Furthermore, the two branches of the dual-path convolution-long short-term memory module have the same structure. Each branch includes multiple cascaded 2D convolutional modules and multiple cascaded long short-term memory layers connected in parallel with the 2D convolutional modules. Each 2D convolutional module includes cascaded 2D convolutional layers, batch normalization layers, modified linear unit activation function layers, and average pooling layers.

[0028] Furthermore, the detection output module includes a cascaded first fully connected layer, a modified linear unit activation function layer, a random deactivation layer, and a second fully connected layer.

[0029] Furthermore, the output of the detection output module is the predicted probability of six types of traffic events: left-side approach, left-side away, right-side approach, right-side away, front, and no category.

[0030] Furthermore, the detection output module uses the softmax activation function to map the output results to continuous values ​​between 0 and 1, and the predicted probabilities of the six types of acoustic traffic events are added together to 1, where 1 indicates that the type of acoustic traffic event is active and 0 indicates that the type of acoustic traffic event is inactive. The category with the highest predicted probability is output as the acoustic non-line-of-sight vehicle approaching and moving away detection results.

[0031] The advantages of this invention are:

[0032] 1. The acoustic non-line-of-sight (NLS) vehicle approach and departure detection method provided by this invention constructs an acoustic NLS vehicle approach and departure detection model, which is composed of a feature extraction module, a dual-path convolutional-long short-term memory (LSTM) module, and a detection output module cascaded together. The training and prediction process of the model does not rely on strong assumptions, avoiding the influence of assumptions about a single acoustic propagation mode on the NLS vehicle localization accuracy. A spectrogram is introduced as a feature representation. The dual-path convolutional-LSTM module adds a second branch to the single-path network and extracts the temporal features of the depth representation through the LSTM network module, enabling the joint acoustic NLS vehicle detection task and localization task, thereby improving the detection accuracy of NLS vehicles and the classification accuracy of NLS vehicle approach and departure states.

[0033] 2. The calculated continuous sound energy distribution results are transformed into a two-dimensional image representation by using the maximum-minimum normalization method, which more effectively realizes the transformation between continuous sound energy distribution and two-dimensional image representation, and the classification accuracy is higher. Attached Figure Description

[0034] The features and advantages of the invention will become more readily apparent from the following description with reference to the accompanying drawings, which are not drawn to scale and some features are enlarged or reduced to show details of specific parts.

[0035] Figure 1 This is a flowchart of an acoustic non-line-of-sight vehicle approach and departure detection method according to an exemplary embodiment of the present invention;

[0036] Figure 2 This is a framework diagram of the acoustic non-line-of-sight vehicle approach and departure detection model in the acoustic non-line-of-sight vehicle approach and departure detection method according to an exemplary embodiment of the present invention;

[0037] Figure 3This is a network structure diagram of the dual-path convolutional-long short-term memory module included in the acoustic non-line-of-sight vehicle approach and departure detection model in the acoustic non-line-of-sight vehicle approach and departure detection method according to an exemplary embodiment of the present invention. Detailed Implementation

[0038] The present invention will now be described in detail with reference to the accompanying drawings and exemplary embodiments thereof. It should be noted that the following detailed description of the present invention is for illustrative purposes only and is not intended to limit the scope of the invention.

[0039] Reference Figure 1 and Figure 2 The acoustic non-line-of-sight vehicle approach and departure detection method, as an exemplary embodiment of the present invention, may include the following steps:

[0040] Step 1: Construct an acoustic non-line-of-sight vehicle approach and departure detection model. The detection model includes a feature extraction module, a dual-path convolutional-long short-term memory (CNN-LSTM) module, and a detection output module.

[0041] The feature extraction module receives multi-channel audio from vehicles passing through the intersection and outputs a spectrogram of the first channel audio data and a continuous sound energy distribution conversion image of the multi-channel audio data.

[0042] The dual-channel convolutional-long short-term memory module receives the spectrogram and continuous acoustic energy distribution transformation image extracted by the feature extraction module, and outputs the temporal and spatial features of the multi-channel audio data.

[0043] The detection output module is used to receive the temporal and spatial features extracted by the dual-channel convolution-long short-term memory module, and output the non-line-of-sight vehicle approach and departure detection results with directional information;

[0044] Step 2: Collect complete multi-channel audio and video data of vehicles passing through the intersection and train the constructed detection model.

[0045] Step 3: Acquire multi-channel audio data of the environment surrounding the intelligent vehicle at the intersection, and input the acquired multi-channel audio data into the trained detection model to obtain the detection results of non-line-of-sight vehicles approaching and moving away.

[0046] The feature extraction module included in the detection model considers the influence of multiple acoustic propagation modes. It does not make assumptions about a single acoustic propagation mode for the received multi-channel audio signals, but directly calculates the output spectrogram and continuous acoustic energy distribution transformation image to avoid the impact of assumptions about a single acoustic propagation mode on the accuracy of non-line-of-sight vehicle positioning. In a specific embodiment, the feature extraction module uses the Short Time Fourier Transform (STFT) method applied to the first channel of the received multi-channel audio to generate the spectrogram; other methods can also be used to generate the spectrogram. Using the spectrogram as the input to the dual-channel convolutional-long short-term memory module effectively solves the problem of lacking acoustic non-line-of-sight vehicle detection feature representation in existing technologies, thereby realizing the detection of other vehicles in the surrounding environment of intelligent driving vehicles.

[0047] In some embodiments of the present invention, the feature extraction module calculates the sound energy distribution results of the continuous surrounding environment from the received multi-channel audio. In particular, the sound energy distribution results of the continuous surrounding environment can be calculated from the received multi-channel audio using maximum controllable response beamforming (SRP-PHAT). Furthermore, the feature extraction module performs maximum-minimum normalization on the continuous sound energy distribution results and converts them into a two-dimensional image representation, which is then output as a continuous sound energy distribution conversion image. This makes the conversion between continuous sound energy distribution and two-dimensional image representation more effective and achieves higher classification accuracy.

[0048] The formula for calculating the pixel value P(i,j) of the continuous sound energy distribution conversion image is as follows:

[0049] P(i,j)=255×(S(i,j)-S min ) / (S max -S min )

[0050] In the formula, S max and S min Let P(i,j) be the maximum and minimum values ​​in the feature matrix of the image size, and let P(i,j) be the pixel value of the continuous acoustic energy distribution transformed image at coordinates (i,j).

[0051] The feature extraction module may include an STFT calculation unit and an SRP-PHAT calculation unit. First, the STFT calculation unit calculates the spectrogram of the first channel audio data. In some embodiments, the first channel audio data is input into the detection network frame by frame. Second, the SRP-PHAT calculation unit calculates the continuous acoustic energy distribution of the multi-channel audio data. The multi-channel audio data is input into the detection network frame by frame. Specifically, a 1-second frame window, 256 points in the Fourier transform, an angular resolution of 3°, and a frequency band of [100, 12000] Hz can be used. The acoustic energy spectrum obtained in each calculation is represented as a (1×61) matrix. 61 acoustic energy spectrum calculation results are continuously generated for a 1-second multi-channel audio sample. These results are combined according to the time dimension to form a (61×61) feature matrix. Then, a continuous acoustic energy distribution transformation image is generated using max-min normalization, with an image size of 61*61.

[0052] The CNN-LSTM module included in the detection model uses the two feature representations mentioned above as input to the classifier. Therefore, a dual-path neural network is constructed, allowing the spectrogram and the continuous acoustic energy distribution transformation image to be used as input simultaneously. Both inputs correspond to the acoustic non-line-of-sight (NLOS) vehicle approach and departure detection results simultaneously, enabling the constructed network to jointly perform the acoustic NLOS vehicle detection and localization tasks. The acoustic NLOS vehicle detection and localization results are matched with each other, improving detection accuracy and avoiding perceptual errors. This dual-path neural network includes a CNN-LSTM module, allowing the feature extraction module to simultaneously extract temporal and spatial features from the feature representation. This enables the network to better learn temporal features, thereby improving detection accuracy.

[0053] like Figure 3 As shown, in an optional embodiment, the two branches of the CNN-LSTM module have the same structure. Each branch includes multiple cascaded 2D convolutional modules and multiple cascaded long short-term memory layers connected in parallel with the 2D convolutional modules. Each 2D convolutional module includes cascaded 2D convolutional layers, batch normalization layers, modified linear unit activation function layers, and average pooling layers.

[0054] The detection output module included in the detection model realizes the mapping from the extracted features to the detection results, including a cascaded first fully connected layer, a modified linear unit activation function layer, a random deactivation layer, and a second fully connected layer.

[0055] In a specific embodiment of the present invention, the output of the detection output module is the predicted probability of six types of traffic events: left-side approach, left-side departure, right-side approach, right-side departure, front, and no class. Specifically, the detection output module can use the softmax activation function to map the output to a continuous value between 0 and 1, and the predicted probabilities of the six types of acoustic traffic events are summed to 1, where 1 indicates that the type of acoustic traffic event is active and 0 indicates that the type of acoustic traffic event is inactive. The category with the highest predicted probability is output as the acoustic non-line-of-sight vehicle approach and departure detection results.

[0056] In the illustrated embodiment, step 2 includes sub-step 2.1, which involves collecting complete multi-channel audio and video data of the vehicle passing through the intersection. Preferably, at intersections with obstructions, an onboard microphone array can be used to simultaneously collect complete audio and video data of the vehicle as it passes. Specifically, a 32-channel microphone array with a forward-facing camera can be used to collect audio and video data. The microphone array is mounted on the top of the intelligent vehicle, and the data collection locations are multiple T-shaped intersections. During collection, the intelligent vehicle is approximately 12 meters away from the intersection, and the vehicle passes through the intersection multiple times from the left and right sides. The microphone array records the complete process of the vehicle passing through the intersection.

[0057] Step 2 includes sub-step 2.2, which divides the collected multi-channel audio segments containing the complete process of a vehicle passing through the intersection into a training set and a validation set. The training and validation sets cannot contain audio data from the same number of passes of the tested vehicles passing through the intersection; therefore, the division is performed on the entire audio segment. The preset ratio for the training and validation sets is unrestricted, but specifically 4:1. Since the training and validation of the detection model are performed using five-fold cross-validation, all recorded complete audio segments are divided into five parts, with no duplicate passes between them.

[0058] Step 2 also includes sub-step 2.3, which involves segmenting and classifying the entire multi-channel audio data of the vehicle passing through the intersection based on the times when the vehicle appears and leaves the field of vision recorded in the acquired video. Multi-channel audio samples are then extracted from the segmented audio segments using a sliding time window. For example, taking the time when the tested vehicle appears in the video image as t0 and the time when it leaves the video image as t1, for each recorded complete audio segment of a vehicle passing through the intersection, the audio segment segmented into [t0-2, t0] is classified as the "nearer" class, the audio segment segmented into [t0, t1] is classified as the "foreign" class, and the audio segment segmented into (t1, t1+2] is classified as the "farther" class. For instance, if a 12-second complete audio segment of a vehicle passing through the intersection from the left is recorded, and according to the synchronously acquired video, the times when the vehicle appears and leaves the field of vision are 4s and 8s respectively, then the audio segment segmented into [2, 4)s is classified as the "left-side closer" class. The audio segment segmented into [4,8]s is classified as the front class, and the audio segment segmented into (8,10]s is classified as the right-side far-away class. For each segmented audio segment, a sliding window with a window length of 1s and a step size of 10ms is used to slide across each segment. The audio data within the time window is used as a sample, and the duration of each audio sample is 1s. For the multi-channel audio segments after classification, a sliding time window with a window length of 1s and a step size of 10ms is used to extract samples. Since the proportion of audio samples in the front class and the near / far-away class is unbalanced, the step size when extracting audio samples in the near / far-away class is shortened to achieve a balance in the sample proportion.

[0059] Step 2 also includes sub-step 2.4, which trains the constructed detection model using the divided training and validation sets.

[0060] As described above, the acoustic non-line-of-sight (NLS) vehicle approach and departure detection method provided by this invention constructs an acoustic NLS vehicle approach and departure detection model, which is composed of a feature extraction module, a dual-path convolutional-long short-term memory (LSTM) module, and a detection output module cascaded together. The training and prediction process of the model does not rely on the strong assumption of a single acoustic propagation mode and can consider the influence of multiple acoustic propagation modes. A spectrogram is introduced as a feature representation. The dual-path convolutional-LSTM module adds a second branch on the basis of the single-path network and extracts the temporal features of the depth representation through the LSTM network module, which enables the joint acoustic NLS vehicle detection task and localization task, thereby improving the detection accuracy of NLS vehicles and the classification accuracy of NLS vehicle approach and departure states.

[0061] The features mentioned and / or shown in the foregoing description of exemplary embodiments of the present invention may be combined in the same or similar manner with one or more other embodiments, combined with features in other embodiments, or substituted for corresponding features in other embodiments. Such combinations or substitutions should also be considered as including within the scope of protection of the present invention.

Claims

1. An acoustic non-line-of-sight vehicle approach and departure detection method, characterized in that, Includes the following steps: Step 1: Construct an acoustic non-line-of-sight vehicle approach and departure detection model. This detection model includes a feature extraction module, a dual-path convolutional-long short-term memory module, and a detection output module. The feature extraction module is used to receive multi-channel audio from vehicles passing through intersections and outputs a spectrogram of the first channel audio data and a continuous sound energy distribution conversion image of the multi-channel audio data. The dual-channel convolutional-long short-term memory module is used to receive the spectrogram and continuous acoustic energy distribution conversion image extracted by the feature extraction module, and output the temporal and spatial features of the multi-channel audio data. The detection output module is used to receive the temporal and spatial features extracted by the dual-channel convolution-long short-term memory module, and output the non-line-of-sight vehicle approach and departure detection results with directional information; Step 2: Collect complete multi-channel audio and video data of vehicles passing through the intersection and train the constructed detection model. Step 3: Acquire multi-channel audio data of the environment surrounding the intelligent vehicle at the intersection, and input the acquired multi-channel audio data into the trained detection model to obtain the detection results of non-line-of-sight vehicles approaching and moving away.

2. The acoustic non-line-of-sight vehicle approach and departure detection method according to claim 1, characterized in that: The feature extraction module calculates the sound energy distribution of the continuous surrounding environment from the received multi-channel audio, and then uses max-min normalization to convert the continuous sound energy distribution into a two-dimensional image representation.

3. The acoustic non-line-of-sight vehicle approach and departure detection method according to claim 2, characterized in that: The feature extraction module uses SRP-PHAT to calculate the sound energy distribution of the continuous surrounding environment from the received multi-channel audio.

4. The acoustic non-line-of-sight vehicle approach and departure detection method according to claim 1 or 2, characterized in that: The feature extraction module uses the STFT method to generate a spectrogram by applying it to the first channel audio of the received multi-channel audio.

5. The acoustic non-line-of-sight vehicle approach and departure detection method according to claim 1 or 2, characterized in that, Step 2 includes the following sub-steps: Step 2.1: Collect complete multi-channel audio and video data of the vehicle passing through the intersection; Step 2.2: Divide the collected multi-channel audio segments containing the complete process of a vehicle passing through an intersection into a training set and a validation set; Step 2.3: Based on the times when the vehicle appears and leaves the field of vision recorded in the video, the entire segment of multi-channel audio data of the vehicle passing through the intersection is segmented and classified. Multi-channel audio samples are extracted from the segmented audio segments through a sliding time window. Step 2.4: Train the constructed detection model using the divided training and validation sets.

6. The acoustic non-line-of-sight vehicle approach and departure detection method according to claim 1 or 2, characterized in that: The two branches of the dual-path convolution-long short-term memory module have the same structure. Each branch includes multiple cascaded 2D convolutional modules and multiple cascaded long short-term memory layers connected in parallel with the 2D convolutional modules. Each 2D convolutional module includes cascaded 2D convolutional layers, batch normalization layers, modified linear unit activation function layers, and average pooling layers.

7. The acoustic non-line-of-sight vehicle approach and departure detection method according to claim 1 or 2, characterized in that: The detection output module includes a cascaded first fully connected layer, a modified linear unit activation function layer, a random deactivation layer, and a second fully connected layer.

8. The acoustic non-line-of-sight vehicle approach and departure detection method according to claim 1 or 2, characterized in that: The output of the detection output module is the predicted probability of six types of traffic events: left-side approach, left-side departure, right-side approach, right-side departure, front, and no category.

9. The acoustic non-line-of-sight vehicle approach and departure detection method according to claim 8, characterized in that: The detection output module uses the softmax activation function to map the output results to continuous values ​​between 0 and 1, and the predicted probabilities of the six types of acoustic traffic events are added together to 1. 1 indicates that the type of acoustic traffic event is active, and 0 indicates that the type of acoustic traffic event is inactive. The category with the highest predicted probability is output as the acoustic non-line-of-sight vehicle approaching and moving away detection results.

Images