Detection method for sensing driving scene and event based on sound

Abstract

The invention provides a detection method for sensing driving scenes and events based on sound, and relates to the technical field of automobile detection. The method comprises the following steps: constructing a sound event relation graph and a driving scene graph based on a traffic scene sound event data set; preprocessing audio data in the sound event data set, extracting phase and amplitude features, and combining the obtained amplitude spectrum and phase spectrum into a feature sequence; constructing a shared acoustic feature information learning network for learning acoustic features of sound events in the audio, and sharing the learned acoustic features to an acoustic scene classification network and a sound event classification network for classification of the sound events and acoustic scenes; constructing a node representation learning module, learning the node representation in each acoustic scene, and then taking the node representation as an advanced node to learn the node representation in the driving scene; and constructing an acoustic scene score output network and a sound event score output network, and realizing final acoustic scene classification and sound event classification tasks according to scores output by the networks.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile detection, and in particular to a detection method for driving scenes and events based on sound perception. Background Art

[0002] With the rapid development of the automotive industry towards intelligent networking, various types of sensor technologies are applied to smart cars, and the perception ability of cars is becoming stronger and stronger. In addition to visual technology, sound signals can also provide a lot of useful information, and have the advantages of not requiring light, and not being disturbed by rainy days and dark days. Audio monitoring equipment using sound event detection technology will be widely used in smart cars in the future due to its low cost, small size, easy and fast installation, strong reliability, not easy to damage, and simple maintenance.

[0003] In recent years, the technical framework of sound event classification and scene classification is mainly based on deep learning, using convolutional neural networks (CNN) and recurrent neural networks (RNN) to build detection models. The features in the audio signal are extracted and classified as the basis for sound event or acoustic scene analysis and detection. However, the model does not consider the sound events with scene characteristics in different scenes as one of the bases for sound scene detection, nor does it use the connection between scenes and events and the connection between events to help the model predict acoustic scenes and sound events. Therefore, there is still a certain gap between the detection ability in complex situations and that of humans.

[0004] Currently, there is no technical solution that can comprehensively extract and utilize the relationship information between acoustic scenes and sound events, as well as the relationship information between sound events, and use this relationship information to perform multi-task joint analysis of sound event detection and acoustic scene classification.

[0005] Noriyuki TONAMI and Keisuke IMOTO pointed out in their paper that there is a close connection between sound event detection (SED) and acoustic scene classification (ASC). However, existing studies often fail to jointly consider the analysis of sound events and acoustic scenes when dealing with SED and ASC problems. Therefore, a new method is proposed to jointly analyze sound events and acoustic scenes by multi-task learning that combines SED and ASC methods. Keisuke Imoto and Seisuke Kyochi pointed out in their paper that the types of sound events occurring in a single scene are limited, and some sound events occur at the same time. Therefore, it is hoped that sound events can be efficiently modeled using limited sound data. The occurrence of sound events is represented as a graph, whose nodes represent the frequency of event occurrence and whose edges represent co-occurring sound events. The graph is used for sound event modeling, and graph Laplace regularization is introduced into the objective function of the neural network.

[0006] Almost all existing technologies only use the acoustic features of each scene in the audio signal picked up by neural network models such as CNN, RNN, CRNN, etc. to perform acoustic scene classification, or only use the independent features of each sound event in the picked up audio signal to perform sound event detection. The connection between acoustic scenes and sound events is not used to help the model make predictions, and the connection between sound events is not considered.

[0007] Existing methods treat acoustic scene classification and sound event detection as two separate detection tasks, without considering or utilizing the relationship between scenes and events and the relationship between sound events, and only detect and analyze them through the acoustic features of the two.

[0008] When constructing a sound event relationship graph, existing methods only consider sound events that occur at the same time, and use the probability of two sound events occurring at the same time to construct an undirected graph or use the conditional probability of two sound events occurring at the same time to construct a directed graph. When constructing an undirected graph, it is impossible to accurately describe the conditional co-occurrence relationship between each sound event and its related sound events. In actual situations, it is very likely that when event A occurs, event B will occur with a high probability, but when event B occurs, the probability of event A occurring is not necessarily high and may be very low. However, existing methods for constructing directed graphs lack descriptions of the frequency of occurrence of various sound events, and the relationship information between scenes and events is insufficient, which may result in the model being unable to distinguish two different scenes with similar or identical sound event types.

[0009] During the driving process, in addition to accurately analyzing and detecting events occurring around the car through sound signals, judging the specific circumstances of the scene, clarifying what scene the vehicle is currently in, and improving the utilization rate of sound information, it is also an important means to further improve the perception and analysis capabilities of the car.

[0010] None of the existing technologies have taken into account the relationship between scenes and events and the relationship between sound events at the same time, nor have they effectively jointly analyzed the two tasks of acoustic scene detection and sound event detection. Currently, there is no technical solution that uses graph neural networks to learn the two relationships, the relationship between scenes and events and the relationship between sound events, and effectively complete the two tasks of acoustic scene detection and sound event detection. Summary of the invention

[0011] The technical problem to be solved by the present invention is to provide a detection method for driving scenes and events based on sound perception in view of the deficiencies of the above-mentioned prior art, so as to realize the detection of driving scenes and events.

[0012] In order to solve the above technical problems, the technical solution adopted by the present invention is: a detection method for driving scenes and events based on sound perception, comprising the following steps:

[0013] Step 1: construct a sound event relationship graph and a driving scene graph based on a traffic scene sound event dataset; the sound event dataset includes audio data and audio tags, and the sound event dataset is divided into a training dataset, a test dataset, and a verification dataset;

[0014] Step 1.1: Obtain label information in the traffic scene sound event dataset and perform sound event statistics;

[0015] Step 1.1.1: Read the label file in the sound event dataset, and obtain the label information of the training dataset in the sound event dataset, including the sound event category in the label file of each audio, and the start and end time of each sound event;

[0016] Each type of sound event is denoted as L i , i = 0, 1, 2, 3...n, n is the total number of sound events, and the start and end time of each sound event is recorded;

[0017] Step 1.1.2: Count the number of occurrences of each type of sound event in the sound event data set, the total number of occurrences of all sound events, and the number of times each type of sound event occurs simultaneously in each audio;

[0018] The situation where two sound events occur simultaneously is recorded as L ij , j = 0, 1, 2, 3 ... n and i ≠ j, L ij Indicates event L i With event L j Occur simultaneously in a piece of audio; according to the sound events with labels in each audio, count the number of occurrences of each type of sound event, recorded as X i , indicating the sound event L i The number of occurrences, X represents the total number of occurrences of various sound events; the number of times two sound events occur at the same time is recorded as X ij Indicates event L i With event L j The number of times it occurs simultaneously in the same audio;

[0019] Step 1.2: Calculate the relationships between various sound events in the sound event dataset and construct a sound event relationship graph for different driving scenarios;

[0020] Step 1.2.1: Calculate the probability of various sound events occurring simultaneously using the prior probability formula;

[0021] Calculate the probability P of various sound events occurring simultaneously ij , according to the number of occurrences of various sound events obtained in step 1.1, X i , and the number of times two sound events occur simultaneously X ij, calculate when the sound event L i When the sound event L occurs j The probability P ij , as shown in the following formula:

[0022] P ij =P(L j |L i )=X ij / X i

[0023] The probability threshold α, α∈(0,1) is used to filter the sound event relationship, and the sound event relationship weights with probability lower than the threshold are reset to 0, as shown in the following formula:

[0024]

[0025] Step 1.2.2: Calculate the occurrence frequency of various sound events in different driving scenarios;

[0026] Use the number of occurrences X of each event obtained in step 1.1 i , and the total number of times each type of sound event occurs X, calculate the frequency of occurrence of each type of sound event in the scene P i :

[0027] P i =X i / X

[0028] Step 1.2.3: Construct the sound event relationship diagram of different driving scenarios;

[0029] Taking various sound events as nodes, the probability of various sound events occurring simultaneously is P ij The sound event relationship graph is constructed as the directed edges connecting these nodes. The occurrence frequency P of each type of sound event i As part of each node's own attributes;

[0030] Step 1.3: Incorporate each sound event relationship graph into the driving scene graph as a high-level node. Each node in the driving scene graph represents a driving scene, and each driving scene has a corresponding sound event relationship graph.

[0031] Step 2: Preprocess the audio data in the sound event dataset, extract phase and amplitude features, and merge the obtained amplitude spectrum and phase spectrum into a feature sequence;

[0032] On the Hanning window, a fast Fourier transform is used to extract a spectrogram from each C channel of the multi-channel audio; then the phase and amplitude of the spectrogram are extracted, and the extracted phase and amplitude are combined as separate features into a feature sequence;

[0033] Step 3: Construct an acoustic scene and sound event joint detection model to detect sound events;

[0034] Step 3.1: Construct a shared acoustic feature information learning network to learn the acoustic features of sound events in the audio, and share the learned acoustic features with the acoustic scene classification network and the sound event classification network;

[0035] The shared acoustic feature information learning network sends the feature sequence output after preprocessing in step 2 as input to the convolutional neural network; in the convolutional neural network part, a three-layer two-dimensional CNN is used to learn the acoustic features in the feature sequence; the first layer of CNN has B1 filters with a 3×3×2C-dimensional receptive field, and the number of filters in the next two layers of CNN is B2=B1 / 2 and B3=B1 / 4 respectively; the step size and padding of the three layers of CNN are both 1 to ensure that the feature sequence length T remains unchanged; after each layer of CNN, the output is standardized using batch normalization and activated using the ReLU function, and then the dimension is reduced using the maximum pooling along the frequency axis, thereby keeping the feature sequence length T unchanged; using a filter kernel spanning all channels, allowing CNN to learn relevant intra-channel features in the time and frequency dimensions; the output of the last CNN layer is T×2×B3;

[0036] Step 3.1.1: Create the first two-dimensional CNN layer. The input of this layer is the feature sequence output by the preprocessing in step 2. The CNN layer uses a filter kernel that spans all channels, allowing the CNN to learn relevant intra-channel features in the time and frequency dimensions. This layer has B1 filters with a 3×3×2C-dimensional receptive field, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; use batch normalization to normalize the convolutional layer output and activate it with the ReLU function, and then use the maximum pooling along the frequency axis to reduce the dimension, thereby keeping the feature sequence length T unchanged;

[0037] Step 3.1.2: Create a second 2D CNN layer with B2 filters of 3×3×2C-dimensional receptive field, and stride and padding of 1 to ensure that the feature sequence length T remains unchanged; the input of this layer is the feature sequence output by the first CNN layer, which first passes through the convolutional layer, and then uses batch normalization to normalize the convolutional layer output and activate it with the ReLU function, and then reduces the dimension after the maximum pooling along the frequency axis;

[0038] Step 3.1.3: Create a third 2D CNN layer with B3 filters of 3×3×2C-dimensional receptive field, stride and padding of 1 to ensure that the sequence length T remains unchanged; the input of this layer is the feature sequence output by the second CNN layer, which first passes through the convolutional layer, then uses batch normalization to normalize the convolutional layer output, activates with the ReLU function, and then reduces the dimension by maximum pooling along the frequency axis;

[0039] Step 3.2: Build a sound event classification network to achieve the task of classifying sound events in audio;

[0040] The sound event classification network includes a bidirectional RNN layer and two FC layers. The feature sequence output in step 3.1.3 is swapped in dimension and further reshaped into a feature vector of length 2B3, and is sent to the bidirectional RNN layer, which is used to learn temporal context information from the CNN output; a gated recurrent unit GRU with Q nodes is used in each RNN layer, and the tanh function is used for activation, and the dimension of the feature sequence output is consistent with the input; the output of the bidirectional RNN layer is then sent to two FC layers to further extract the acoustic features of the sound event; the first FC layer contains R nodes, each of which has linear activation; the second FC layer consists of N nodes, and the value of N is the number of sound event categories in the corresponding sound data set;

[0041] Step 3.2.1: Swap the dimension of the feature sequence output by CNN and further reshape it into a feature vector of length 2B3;

[0042] Step 3.2.2: Create two bidirectional RNN layers and feed the feature sequence reshaped in step 3.1.4 into them. Each RNN layer uses a gated recurrent unit (GRU) with Q nodes to extract the temporal context information of the sound event and activates it using the tanh function. The dimension of the feature sequence output is consistent with the input.

[0043] Step 3.2.3: Create two FC layers and feed the feature sequence of the sound event temporal context information extracted by the bidirectional RNN layer into them for further training;

[0044] The first FC layer contains R nodes, each with linear activation;

[0045] The second FC layer consists of N nodes, where the value of N is the number of sound event categories in the corresponding dataset;

[0046] Step 3.3: Construct an acoustic scene classification network to classify acoustic scenes;

[0047] The feature sequence output from step 3.1.3 is sent as input to the convolutional neural network; in the convolutional neural network part, a two-layer two-dimensional CNN is used to learn the acoustic features in the feature sequence; the first CNN layer has B4 filters with a 3×3×2C-dimensional receptive field, and the number of filters in the second CNN layer is also B4; the step size and padding of the two CNN layers are both 1 to ensure that the feature sequence length T remains unchanged; after each layer of CNN, the output is standardized using batch normalization and activated using the ReLU function to keep the feature sequence length T unchanged; the feature sequence output from the second layer of CNN is exchanged in dimension and further reshaped before being sent to two FC layers to further extract the acoustic features of the acoustic scene; the first FC layer contains R nodes, each of which has linear activation; the second FC layer consists of M nodes, and the value of M is the number of sound scenes in the corresponding acoustic dataset;

[0048] Step 3.3.1: Create the first two-dimensional CNN layer. The input of this layer is the feature sequence output by step 3.1.3. Use the filter kernel across all channels to allow CNN to learn relevant intra-channel features in the time and frequency dimensions. This layer has B4 filters with a 3×3×2C-dimensional receptive field, and the step size and padding are both 1 to ensure that the sequence length T remains unchanged; use batch normalization to standardize the output and use the ReLU function for activation.

[0049] Step 3.3.2: Create a second 2D CNN layer with B5 filters of 3×3×2C-dimensional receptive field, stride and padding of 1 to ensure that the sequence length T remains unchanged; the input of this layer is the feature sequence output by the first CNN layer, which first passes through the convolutional layer, then the output is normalized using batch normalization and activated using the ReLU function;

[0050] Step 3.3.3: The feature sequence output by the second CNN layer is swapped and further reshaped into a T frame sequence and then fed into two FC layers to further extract the acoustic features of the acoustic scene;

[0051] Step 3.3.4: Create two FC layers; the first FC layer contains R nodes, each with linear activation;

[0052] The second FC layer consists of M nodes, where the value of M is the number of sound scenes in the corresponding dataset;

[0053] Step 3.4: Build a node representation learning module to learn the node representation in each acoustic scene and then use it as a high-level node to learn the node representation in the driving scene;

[0054] The node representation learning module uses the GCN network to learn the node representation in each acoustic scene respectively and then uses it as a high-level node to learn the node representation in the driving scene; the GCN network uses two GCN layers, each GCN layer converts the node representation H learned by the previous layer into (l) As input, the new node representation H is output after learning. (l+1) ; The input node representation of the first GCN layer is the word embedding of the label, where N represents the number of labels, that is, the number of types of sound events in the sound dataset, and f represents the dimension of the label word embedding; for the last layer, the output is o represents the feature dimension of the output feature sequence, which is the same as the feature sequence dimension output by step 3.2 and step 3.3; the node representation output of multiple acoustic scenes is embedded as the node of the driving scene It is sent to the GCN network again, where M represents the number of acoustic scene types in the driving scene, and F is the embedding dimension of the high-level nodes representing the acoustic scene features; after two layers of GCN, the final output is the feature sequence represented by the node of the driving scene

[0055] Step 3.4.1: Obtain parameters based on the sound event relationship graph structure constructed in step 1, graph G = (V, E), where V contains all N nodes in the sound event relationship network, that is, the N types of sound events to be detected, v i ∈V, E represents the edge between nodes (v i ,v i )∈E; the eigenvector matrix of the node is Where N is the number of nodes and f is the dimension of the feature vector;

[0056] Step 3.4.2: In order to retain the node's own information while aggregating neighbor information, add self-connections to each node:

[0057]

[0058] Among them, A represents the adjacency matrix of graph G, The elements in are weighted real numbers, I N Indicates that each node has added a self-connection, N is the number of nodes, It means that each node has the adjacency matrix after adding the self-connection relationship. Represents the degree matrix after each self-connection is added, the degree matrix D = ∑ j A ij ;

[0059] Step 3.4.3: Use parameter β to adjust the weight ratio of the node itself and its neighbors. The adjacency matrix after adjusting the weight ratio of itself and its neighbors is as follows:

[0060]

[0061] in, Represents the adjacency matrix after adjusting the weight ratio between itself and its neighbors. β is used to allocate the weight ratio between neighbors and itself, β∈[0,1];

[0062] Step 3.4.4: Improve the propagation of the GCN network between layers;

[0063] The improved GCN network propagates between layers according to the following formula:

[0064]

[0065] Among them, it means that H is adjusted (l) represents the input of the l-th layer GCN network, W(l) represents the parameters to be trained, and h() represents the corresponding activation function;

[0066] Step 3.4.5: Construct the first GCN layer and input the node feature vector matrix H(0) into the GCN network improved by the above step 3.4.4;

[0067]

[0068] Among them, H(1) is the node feature vector matrix after aggregation and update of the first GCN layer;

[0069] Step 3.4.6: Construct the second GCN layer and input the node feature vector matrix H(1) after aggregation and update by the first GCN layer into it. The node feature vector matrix after aggregation and update is as shown in the following formula:

[0070]

[0071] Among them, H(2) is the node feature vector matrix after aggregation and update of the second GCN layer;

[0072] The node feature vector matrix H(2) learned in the last layer is used as the node representation output of the acoustic scene;

[0073] Step 3.4.7: Embed the node representation output of multiple acoustic scenes in step 3.4.6 as the node of the driving scene It is sent to the GCN network again, where M represents the number of acoustic scene types in the driving scene, and F is the embedding dimension of the high-level nodes representing the acoustic scene features;

[0074] After two layers of GCN, the final output is the feature sequence of the node representation of the driving scene

[0075] Step 3.5: Transpose the feature sequence represented by the node of the driving scene output in step 3.4.7; perform matrix multiplication on the feature sequence before transposition and the acoustic feature sequence of sound event detection to obtain a combined first feature sequence; perform corresponding element multiplication on the feature sequence after transposition and the acoustic feature sequence of acoustic scene detection to obtain a combined second feature sequence;

[0076] Step 3.6: Construct an acoustic scene score output network and a sound event score output network, and implement the final acoustic scene classification and sound event classification tasks based on the scores output by the networks;

[0077] The two combined feature sequences in step 3.5 are respectively sent to the acoustic scene score output network and the sound event score output network. Both the acoustic scene score output network and the sound event score output network are implemented by the FC layer; the FC layer of the acoustic scene score output network consists of M nodes with sigmoid activation, each node corresponds to one of the M acoustic scene categories to be detected; the FC layer of the sound event score output network consists of N nodes with sigmoid activation, each node corresponds to one of the N sound event categories to be detected; using sigmoid activation, multiple classes can be activated at the same time, and scores are output, and the cross entropy loss function is used for training, and the final classification and detection results are output. The outputs of ASC and SED are both in the continuous range of [0,1]. When the output exceeds the threshold value of 0.5, it is determined that the vehicle is in the acoustic scene or the event exists. For the scene determined to be in, the category of the scene is output, and for the sound event determined to exist, the start and end time and duration of the event are output;

[0078] Step 3.6.1: Create an FC layer for acoustic scene detection, and feed the feature sequence combined in step 3.5 into it. The FC layer consists of M nodes with sigmoid activation, each node corresponds to one of the M acoustic scene categories to be detected; using sigmoid activation, multiple classes can be activated at the same time and scores can be output; use the cross entropy loss function for training, and output the final acoustic scene classification result. The acoustic scene detection output is in the continuous range of [0,1]. When the output exceeds the threshold of 0.5, it is determined that the vehicle is in the acoustic scene;

[0079] Step 3.6.2: Create an FC layer for sound event detection, and feed the first feature sequence M×N combined in step 3.5 into it. The FC layer consists of N nodes with sigmoid activation, each node corresponds to one of the N sound event categories to be detected, and the output of the FC layer is M×N; using sigmoid activation, multiple classes can be activated at the same time and scores can be output; use the cross entropy loss function for training, and output the final sound classification result. The sound event detection output is in the continuous range of [0,1]. When the output exceeds the threshold of 0.5, it is determined that the sound event exists;

[0080] Step 3.6.3: Output the acoustic scene and sound event category as well as the start and end time and duration of the sound event determined in steps 3.6.1 and 3.6.2.

[0081] The beneficial effect of adopting the above technical solution is that: the present invention provides a detection method for driving scenes and events based on sound perception, which extracts two kinds of relationship information: the connection between acoustic scenes and sound events during driving and the connection between sound events, and constructs a corresponding graph structure, and uses a multi-layer graph structure to integrate the relationship between sound events and the relationship between acoustic scenes and sound events into one graph. The two kinds of relationship information are learned using a graph neural network and used in the multi-task joint detection and analysis of acoustic scenes and sound events, so that both acoustic scene detection and sound event detection can obtain more prior knowledge, effectively improving the detection capabilities of both. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A flow chart of a method for detecting driving scenes and events based on sound perception provided by an embodiment of the present invention;

[0083] Figure 2 A diagram showing the relationship between sound events in different scenarios provided by an embodiment of the present invention, wherein (a) is a highway scenario, and (b) is a country road scenario;

[0084] Figure 3 A driving scene diagram provided by an embodiment of the present invention;

[0085] Figure 4 A flowchart of audio data preprocessing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0086] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0087] In this embodiment, a detection method for driving scenes and events based on sound perception is provided. Figure 1As shown, the following steps are included:

[0088] Step 1: construct a sound event relationship graph and a driving scene graph based on an existing public traffic scene sound event dataset; the sound event dataset includes audio data and audio tags, and the sound event dataset is divided into a training dataset, a test dataset, and a verification dataset;

[0089] Step 1.1: Obtain label information in the traffic scene sound event dataset and perform sound event statistics;

[0090] Step 1.1.1: Read the label file in the sound event dataset, and obtain the label information of the training dataset in the sound event dataset, including the sound event category in the label file of each audio, and the start and end time of each sound event;

[0091] Each type of sound event is denoted as L i , i = 0, 1, 2, 3...n, n is the total number of sound events, and the start and end time of each sound event is recorded;

[0092] Step 1.1.2: Count the number of occurrences of each type of sound event in the sound event data set, the total number of occurrences of all sound events, the number of simultaneous occurrences of each type of sound event in each audio, and the number of consecutive occurrences;

[0093] The situation where two sound events occur simultaneously is recorded as L ij , j = 0, 1, 2, 3 ... n and i ≠ j, L ij Indicates event L i With event L j Occur simultaneously in a piece of audio; according to the sound events with labels in each audio, count the number of occurrences of each type of sound event, recorded as X i , indicating the sound event L i The number of occurrences, X represents the total number of occurrences of various sound events; the number of times two sound events occur at the same time is recorded as X ij Indicates event L i With event L j The number of times it occurs simultaneously in the same audio;

[0094] Step 1.2: Calculate the relationships between various sound events in the sound event dataset and construct a sound event relationship graph for different driving scenarios;

[0095] Step 1.2.1: Calculate the probability of various sound events occurring simultaneously and successively using the prior probability formula;

[0096] Calculate the probability P of various sound events occurring simultaneously ij , according to the number of occurrences of various sound events obtained in step 1.1, Xi , and the number of times two sound events occur simultaneously X ij , calculate when the sound event L i When the sound event L occurs j The probability P ij , as shown in the following formula:

[0097] P ij =P(L j |L i )=X ij / X i

[0098] It should be noted that P ij With P ji They are not necessarily equivalent and their meanings are also different.

[0099] Since there are some differences in the probabilities of various events between the training data and the data obtained in actual use, in order to avoid some interference relationships with very low probabilities calculated using the training data from having an adverse effect on the model test, and to avoid the model from overfitting the graph structure generated by the training set during training, a probability threshold α, α∈(0,1) is used to filter the sound event relationships, and the weights of the sound event relationships with probabilities below the threshold are reset to 0, as shown in the following formula:

[0100]

[0101] This operation can effectively reduce possible interference information.

[0102] Step 1.2.2: Calculate the occurrence frequency of various sound events in different driving scenarios;

[0103] Use the number of occurrences X of each event obtained in step 1.1 i , and the total number of times each type of sound event occurs X, calculate the frequency of occurrence of each type of sound event in the scene P i :

[0104] P i =X i / X

[0105] Step 1.2.3: Construct the sound event relationship diagram of different driving scenarios;

[0106] Taking various sound events as nodes, the probability of various sound events occurring simultaneously is P ij The sound event relationship graph is constructed as the directed edges connecting these nodes. The occurrence frequency P of each type of sound event i As part of each node's own attributes;

[0107] Step 1.3: Incorporate each sound event relationship graph into the driving scene graph as a high-level node. Each node in the driving scene graph represents a driving scene, and each driving scene has a corresponding sound event relationship graph.

[0108] After the above steps, the sound event relationship graph structure is constructed. Some examples of the constructed graphs are as follows: Figure 2 As shown, the edges represent the relationship between the simultaneous occurrence of sound events, and the numbers represent the probabilities. This embodiment takes some sounds picked up by a car in two scenes, a highway and a country road, as examples. The sound data includes the sound of a vehicle driving, a horn, a violent collision sound of a car accident, the sound of brakes, talking sounds, and the sounds of animals.

[0109] The following are examples of the meaning of the icons: Figure 2 The violent collision sound of a car accident in the highway scene has an edge pointing to the whistle sound and an edge pointing to the brake sound. At the same time, the violent collision sound of a car accident has a number, and each edge has another number, which means that when the violent collision sound of a car accident occurs, there is a 52% probability that the whistle sound will also appear, and there is an 81% probability that the brake sound will also appear, while the probability of the violent collision sound of a car accident is 2%, the probability of the whistle sound is 9%, and the probability of the brake sound is 11%. The meaning of various data in the rural road scene is the same.

[0110] Finally, these small graphs constructed for each acoustic scene are incorporated into the large graph of the driving scene as a high-level node, such as Figure 3 As shown, each node in the large graph represents a driving scene, and each driving scene has a corresponding sound event relationship graph.

[0111] Step 2: Preprocess the audio data in the sound event dataset, such as Figure 4 As shown, the phase and amplitude features are extracted, and the obtained amplitude spectrum and phase spectrum are combined into a feature sequence;

[0112] A 512-point fast Fourier transform (FFT) is used to extract a spectrogram from each C channel of the multi-channel audio on a Hanning window with a length of 512 and an overlap of 50%. The phase and amplitude of the spectrogram are then extracted and used as separate features. In order to improve the efficiency of audio signal processing and simplify model design, only half of the positive frequencies are used when extracting the phase and amplitude information of the spectrogram, and the zeroth bit is not used. The choice of this method is based on the symmetry of the negative frequency part and the positive frequency part. The two contain the same information but opposite signs. Therefore, in this way, the amount of calculation and storage requirements can be effectively reduced, the calculation efficiency can be improved, and the information of the negative frequency part can be retained. Figure 4The preprocessed feature vector is a feature sequence of T frames, with an overall size of T×256×2C, where 2C is composed of C amplitude and C phase components.

[0113] Step 2.1: Set the audio sampling frequency to 44.1KHz, divide the audio data into frames, set the frame length T to 512 sampling points, and the frame shift J to half of the frame length, that is, 256 sampling points. The audio data collected at the k frame is shown in the following formula:

[0114] x k (m) = x[kJ:KJ+M-1]

[0115] Where k∈[0,K] is the frame index value, indicating the kth frame of the audio data, and K is the total number of frames. Indicates the audio data collected at frame k, 1 <m<M;

[0116] Step 2.2: Performing a windowing operation frame by frame on the frame-processed audio data;

[0117] Step 2.2.1: Select Hanning window as the window function;

[0118] The Hanning window function is shown in the following formula:

[0119]

[0120] Among them, w(l win ) represents the Hanning window function, a is the coefficient, L win is the window length; if a is set to 0.5, the Hanning window will be obtained, and the formula can be simplified to

[0121]

[0122] Step 2.2.2: performing a windowing operation frame by frame on the framed audio data obtained in step 2.1;

[0123] y k (m) = x k (m)w(l win )

[0124] Among them, x k (m) represents the audio data to be windowed, Represents the kth frame data of the windowed audio data in the time domain;

[0125] Step 2.3 obtains the spectrum of the audio data and extracts the phase and amplitude;

[0126] Step 2.3.1: Convert the framed and windowed audio data from the time domain to the frequency domain through fast Fourier transform to obtain the frequency domain signal of the audio data;

[0127]

[0128] in, is a complex number, representing the audio data after fast Fourier transform, y k (m) is the mth sample in the original signal sequence, M is the length of the signal sequence, l is the frequency variable in the frequency domain, ranging from 0 to M-1, j is the imaginary unit, e -j2πlm / M It is the complex exponential term in Fourier transform, which indicates the rotation factor; the real part indicates the lth frequency band in the kth frame m. fft The imaginary part represents the amplitude of the frequency component in the mth frequency band of the kth frame. fft The phase of the frequency component, L fft is the length of the fast Fourier transform;

[0129] Step 2.3.2: Perform a modulo operation on the frequency domain signal obtained in step 2.3.1 to obtain the amplitude spectrum U of the audio data k (l fft );

[0130] U k (l fft )=|S k (l fft )|

[0131] Step 2.3.3: Perform a phase angle operation on the frequency domain signal obtained in step 2.3.1 to obtain the phase spectrum θ of the audio data k (l fft );

[0132] θ k (l fft )=argtan(S k (l fft ))

[0133] Step 2.3.4: Combine the amplitude spectrum and phase spectrum obtained in step 2.3.2 and step 2.3.3 into a feature sequence with a shape of 512×256×2C; this embodiment uses four microphones to receive the audio signal outside the car, so C is 4, where the 2C size consists of C amplitude and C phase components. The final shape size of the feature sequence is 512×256×8;

[0134] Step 3: Construct an acoustic scene and sound event joint detection model to detect sound events;

[0135] Step 3.1: Construct a shared acoustic feature information learning network to learn the acoustic features of sound events in the audio, and share the learned acoustic features with the acoustic scene classification network and the sound event classification network;

[0136] The shared acoustic feature information learning network, such as Figure 1 As shown in the figure, the feature sequence output after preprocessing in step 2 is sent as input to the convolutional neural network; in the convolutional neural network part, a three-layer two-dimensional CNN is used to learn the acoustic features in the feature sequence; the first layer of CNN has B1 filters with a 3×3×2C-dimensional receptive field, B1 is 256, C is 4, and the number of filters in the next two layers of CNN is 128 and 64 respectively; the step size and padding of the three layers of CNN are both 1 to ensure that the feature sequence length T remains unchanged; after each layer of CNN, the output is standardized using batch normalization and activated using the ReLU function, and then the dimension is reduced using the maximum pooling along the frequency axis to keep the feature sequence length T unchanged; using a filter kernel that spans all channels allows CNN to learn relevant intra-channel features in the time and frequency dimensions; the output of the last CNN layer is T×2×B3, that is, 512×2×64, where the frequency dimension is reduced to 2, which is the effect of the maximum pooling layer; after each of the three CNN layers, the maximum pooling on the frequency is 8, 4, and 4 respectively. The specific steps are as follows:

[0137] Step 3.1.1: Create the first two-dimensional CNN layer. The input of this layer is the feature sequence 512×256×8 output by step 2 preprocessing. The CNN layer uses a filter kernel across all channels to allow CNN to learn relevant intra-channel features in the time and frequency dimensions. This layer has B1 filters with a 3×3×2C-dimensional receptive field, B1 is 256, C is 4, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; after the convolution layer, the feature sequence becomes 512×256×256; use batch normalization to standardize the convolution layer output and activate it with the ReLU function, then use the maximum pooling 1×8 along the frequency axis to reduce the dimension to 512×32×256, thereby keeping the feature sequence length T unchanged;

[0138] Step 3.1.2: Create a second two-dimensional CNN layer, which has B2 filters with 3×3×8 dimensional receptive fields, B2 is 128, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; the input of this layer is the feature sequence 512×32×256 output by the first CNN layer. It first passes through the convolution layer, and the dimension of the feature sequence becomes 512×32×128 after convolution. Then, the convolution layer output is standardized using batch normalization and activated using the ReLU function. After that, the dimension is reduced to 512×8×128 after the maximum pooling along the frequency axis, and the maximum pooling on the frequency is 1×4;

[0139] Step 3.1.3: Create the third two-dimensional CNN layer, which has B3 filters with 3×3×8 dimensional receptive fields, B3 is 64, and the step size and padding are both 1 to ensure that the sequence length T remains unchanged; the input of this layer is the feature sequence 512×8×128 output by the second CNN layer. It first passes through the convolution layer, and the dimension of the feature sequence becomes 512×8×64 after convolution. Then, the convolution layer output is standardized using batch normalization and activated using the ReLU function. After that, the dimension is reduced to 512×2×64 through the maximum pooling along the frequency axis, and the maximum pooling on the frequency is 1×4;

[0140] Step 3.2: Build a sound event classification network to achieve the task of classifying sound events in audio;

[0141] The sound event classification network is as follows Figure 1 As shown, it includes a bidirectional RNN layer and two FC layers. The feature sequence 512×2×64 output in step 3.1.3 is swapped in dimension and further reshaped into a feature vector of length 2B3, i.e., 512×128, and sent to the bidirectional RNN layer. The bidirectional RNN layer is used to learn temporal context information from the CNN output; in each RNN layer, a gated recurrent unit (GRU) with Q nodes is used, the value of Q is 128, and the tanh function is used for activation. The dimension of the feature sequence output is consistent with the input; the output of the bidirectional RNN layer is then sent to two FC layers to further extract the acoustic features of the sound event; the first FC layer contains R nodes, R is 128, and each node has linear activation; the second FC layer consists of N nodes, and the value of N is the number of sound event categories in the corresponding sound data set; the feature sequence of the acoustic features output by the final sound event classification network is 512×N. The specific steps are as follows:

[0142] Step 3.2.1: The feature sequence 512×2×64 output by CNN is swapped into 2×512×64 and further reshaped into a feature vector of length 2B3, i.e. 512×128.

[0143] Step 3.2.2: Create two bidirectional RNN layers and feed the feature sequence reshaped in step 3.1.4 into them. Each RNN layer uses a gated recurrent unit (GRU) with Q nodes to extract the temporal context information of the sound event (i.e., obtain the temporal information (temporal relationship) of various sound events in the audio). Q is 128 and activated by the tanh function. The dimension of the feature sequence output is consistent with the input, i.e., 512×128.

[0144] Step 3.2.3: Create two FC layers and feed the feature sequence of the sound event temporal context information extracted by the bidirectional RNN layer into them for further training;

[0145] The first FC layer contains R nodes, R is 128, and each node has linear activation;

[0146] The second FC layer consists of N nodes, where N is the number of sound event categories in the corresponding dataset; the feature sequence of the final output acoustic features is 512×N;

[0147] Step 3.3: Construct an acoustic scene classification network to classify acoustic scenes;

[0148] like Figure 1 As shown, the feature sequence 512×2×64 output from step 3.1.3 is sent as input to the convolutional neural network; in the convolutional neural network part, a two-layer two-dimensional CNN is used to learn the acoustic features in the feature sequence; the first CNN layer has B4 filters with 3×3×2C-dimensional receptive fields, B4 is 64, C is 4, and the number of filters in the second CNN layer is 64; the step size and padding of the two CNN layers are 1 to ensure that the feature sequence length T remains unchanged; after each layer of CNN, batch normalization is used to standardize the output and ReLU function is used for activation to keep the feature sequence length T unchanged; filters across all channels are used The filter kernel allows CNN to learn relevant intra-channel features in the time and frequency dimensions; the output of the second layer of CNN is T×2×B5, that is, 512×2×64; the output feature sequence 512×2×64 is swapped and further reshaped into a feature vector of length 2B5, that is, 512×128, and sent to two FC layers to further extract the acoustic features of the acoustic scene; the first FC layer contains R nodes, R is 128, and each node has a linear activation; the second FC layer consists of M nodes, and the value of M is the number of sound scenes in the corresponding acoustic dataset; the final output feature sequence of acoustic features is 512×M. The specific steps are as follows:

[0149] Step 3.3.1: Create the first two-dimensional CNN layer. The input of this layer is the feature sequence 512×2×64 output from the preprocessing in step 2. The filter kernel across all channels is used to allow CNN to learn relevant intra-channel features in the time and frequency dimensions. This layer has B4 filters with a 3×3×2C-dimensional receptive field, B4 is 64, C is 4, and the step size and padding are both 1 to ensure that the sequence length T remains unchanged. After the convolution layer, the feature sequence is 512×2×64. Use batch normalization to standardize the output and activate it with the ReLU function.

[0150] Step 3.3.2: Create a second 2D CNN layer with B5 filters of 3×3×8 dimensional receptive field, B5 is 64, and the step size and padding are both 1 to ensure that the sequence length T remains unchanged. The input of this layer is the feature sequence 512×2×64 output by the first CNN layer. It first passes through the convolution layer, and the feature sequence dimension after convolution is 512×2×64. Then the output is standardized using batch normalization and activated using the ReLU function.

[0151] Step 3.3.3: The feature sequence 512×2×64 output by CNN is swapped into 2×512×64 and further reshaped into a T-frame sequence of feature vectors of length 2B5, i.e. 512×128.

[0152] Step 3.3.4: Create two FC layers; the first FC layer contains R nodes, R is 128, and each node has a linear activation.

[0153] The second FC layer consists of M nodes, where M is the number of sound scenes in the corresponding dataset. The feature sequence of the final output acoustic features is 512×M.

[0154] Step 3.4: Build a node representation learning module to learn the node representation in each acoustic scene and then use it as a high-level node to learn the node representation in the driving scene;

[0155] The process is as follows: use GCN to learn the node representation in each acoustic scene separately and then use it as a high-level node to learn the node representation in the driving scene; Figure 1 As shown, two GCN layers are used, each of which represents the node H learned by the previous layer. (l) As input, the new node representation H(l+1) is output after learning; the input node representation of the first GCN layer is the word embedding of the label, where N represents the number of labels, that is, the number of types of sound events in the sound dataset, and f represents the dimension of the label word embedding; for the last layer (i.e. the second layer), the output is o represents the feature dimension of the output feature sequence, which is the same as the feature sequence dimension output in step 3.2 and step 3.3, which is 512; the node representation output of multiple acoustic scenes is used as the node embedding of the driving scene It is sent to GCN again, where M represents the number of acoustic scene types in the driving scene, and F is the embedding dimension of the high-level nodes representing the acoustic scene features; after two layers of GCN, the final output is the feature sequence represented by the node of the driving scene

[0156] Step 3.4.1: Obtain parameters based on the sound event relationship graph structure constructed in step 1, graph G = (V, E), where V contains all N nodes in the sound event relationship network, that is, the N types of sound events to be detected, v i ∈V, E represents the edge between nodes (v i ,v i )∈E; degree matrix; the eigenvector matrix of the node is Where N is the number of nodes and f is the dimension of the feature vector;

[0157] Step 3.4.2: In order to retain the node's own information while aggregating neighbor information, add self-connections to each node:

[0158]

[0159] Among them, A represents the adjacency matrix of graph G, The elements in are weighted real numbers, I N Indicates that each node has added a self-connection, N is the number of nodes, It means that each node has the adjacency matrix after adding the self-connection relationship. Represents the degree matrix after each self-connection is added, the degree matrix D = ∑ j A ij ;

[0160] Step 3.4.3: Due to the problem of over-smoothing in the training of multi-layer graph neural networks, the distinguishability of nodes will be reduced. In order to obtain better node representation, reduce the impact of over-smoothing, and maintain sufficient receptive field, this application uses parameter β to adjust the weight ratio of the node itself and its neighbors. The adjacency matrix after adjusting the weight ratio of itself and its neighbors is shown in the following formula:

[0161]

[0162] in, Represents the adjacency matrix after adjusting the weight ratio between itself and its neighbors. β is used to allocate the weight ratio between neighbors and itself, β∈[0,1]; the larger the value of β, the greater the weight of neighbors, and the smaller the value of β, the greater the weight of itself.

[0163] Step 3.4.4: Improve the propagation of the GCN network between layers;

[0164] The propagation between layers of the ordinary GCN network follows the following formula:

[0165]

[0166] in, It means that each node has the adjacency matrix after adding the self-connection relationship. Represents the degree matrix after each self-connection is added, H (l) represents the input of the l-th layer GCN network, W (l) Represents the parameters to be trained, and h() represents the corresponding activation function.

[0167] The improved GCN network propagates between layers according to the following formula:

[0168]

[0169] in, Represents the adjacency matrix after adjusting the weight ratio between itself and its neighbors.

[0170] Step 3.4.5: Construct the first GCN layer and transform the node feature vector matrix H (0) Input into the GCN network improved by the above step 3.4.4;

[0171]

[0172] Among them, H (1) Aggregate the updated node feature vector matrix for the first GCN layer;

[0173] Step 3.4.6: Construct the second GCN layer and aggregate the updated node feature vector matrix H after the first GCN layer (1) Input the node feature vector matrix after aggregation update, as shown in the following formula:

[0174]

[0175] Among them, H (2) Aggregate the updated node feature vector matrix for the second GCN layer;

[0176] The node feature vector matrix H learned in the last layer (2) The node representation output is used as the acoustic scene;

[0177] Step 3.4.7: Embed the node representation output of multiple acoustic scenes in step 3.4.6 as the node of the driving scene It is sent to the two-layer GCN again, where M represents the number of acoustic scene types in the driving scene, and F is the embedding dimension of the high-level nodes representing the acoustic scene features;

[0178] After two layers of GCN, the final output is the feature sequence of the node representation of the driving scene It is M×512.

[0179] Step 3.5: Transpose the feature sequence M×512 represented by the node of the driving scene output in step 3.4.7 to a feature sequence 512×M; perform matrix multiplication on the feature sequence M×512 before transposition and the acoustic feature sequence 512×N of sound event detection to obtain a combined feature sequence M×N; perform corresponding element multiplication on the feature sequence 512×M after transposition and the acoustic feature sequence of acoustic scene detection to obtain a combined feature sequence 512×M;

[0180] Step 3.6: Construct an acoustic scene score output network and a sound event score output network, and implement the final acoustic scene classification and sound event classification tasks based on the scores output by the networks;

[0181] The process is as follows: The two combined feature sequences in step 3.5 are sent to the acoustic scene score output network and the sound event score output network respectively, such as Figure 1 As shown, two FC layers are used as the score output networks of the two respectively; the FC layer of the acoustic scene consists of M nodes with sigmoid activation, each node corresponds to one of the M acoustic scene categories to be detected; the FC layer of the sound event consists of N nodes with sigmoid activation, each node corresponds to one of the N sound event categories to be detected; using sigmoid activation, multiple classes can be activated at the same time, and scores are output, and the cross entropy loss function is used for training, and the final classification and detection results are output. The outputs of ASC and SED are both in the continuous range of [0,1]. When the output exceeds the threshold of 0.5, it is determined that the vehicle is in the acoustic scene or the event exists. For the scene determined to be in, the category of the scene is output, and for the sound event determined to exist, the start and end time and duration of the event are output;

[0182] Step 3.6.1: Create an FC layer for acoustic scene detection, and feed the feature sequence 512×M combined in step 3.5 into it. The FC layer consists of M nodes with sigmoid activation, each node corresponds to one of the M acoustic scene categories to be detected, and the output of the FC layer is 512×M; using sigmoid activation, multiple classes can be activated at the same time and scores can be output; use the cross entropy loss function for training, and output the final acoustic scene classification result. The ASC output is in the continuous range of [0,1]. When the output exceeds the threshold of 0.5, it is determined that the vehicle is in the acoustic scene;

[0183] Step 3.6.2: Create an FC layer for sound event detection, and feed the feature sequence M×N combined in step 3.5 into it. The FC layer consists of N nodes with sigmoid activation, each node corresponds to one of the N sound event categories to be detected, and the output of the FC layer is M×N; using sigmoid activation, multiple classes can be activated at the same time and scores can be output; use the cross entropy loss function for training, and output the final sound classification result. The SED output is in the continuous range of [0,1]. When the output exceeds the threshold of 0.5, the sound event is determined to exist;

[0184] Step 3.6.3: Output the acoustic scene and sound event category as well as the start and end time and duration of the sound event determined in steps 3.6.1 and 3.6.2.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A detection method for driving scenes and events based on sound perception, Features: The following steps are involved: Step 1: construct a sound event relationship graph and a driving scene graph based on a traffic scene sound event dataset; the sound event dataset includes audio data and audio tags, and the sound event dataset is divided into a training dataset, a test dataset, and a verification dataset; Step 2: Preprocess the audio data in the sound event dataset, extract phase and amplitude features, and merge the obtained amplitude spectrum and phase spectrum into a feature sequence; Step 3: Construct an acoustic scene and sound event joint detection model to detect sound events; Step 3.1: Construct a shared acoustic feature information learning network to learn the acoustic features of sound events in the audio, and share the learned acoustic features with the acoustic scene classification network and the sound event classification network; The shared acoustic feature information learning network sends the feature sequence output after the preprocessing in step 2 as input to the convolutional neural network; in the convolutional neural network part, a three-layer two-dimensional CNN is used to learn the acoustic features in the feature sequence; The first layer of CNN has B 1 The number of filters in the next two layers of CNN is B. 2 =B 1 / 2,B 3 =B 1 / 4; the step size and padding of the three-layer CNN are both 1 to ensure that the feature sequence length T remains unchanged; After each CNN layer, the output is normalized using batch normalization and activated using the ReLU function, and then the dimension is reduced using the maximum pooling along the frequency axis, thereby keeping the feature sequence length T unchanged; using a filter kernel spanning all channels allows CNN to learn relevant intra-channel features in the time and frequency dimensions; the output of the last CNN layer is a feature sequence T×2×B 3 ; Step 3.2: Build a sound event classification network to achieve the task of classifying sound events in audio; The sound event classification network includes a bidirectional RNN layer and two FC layers. The feature sequence output in step 3.1 is swapped and further reshaped into a length of 2B. 3 The feature vector is sent to the bidirectional RNN layer, which is used to learn the temporal context information from the CNN output. In each RNN layer, a gated recurrent unit GRU with Q nodes is used and activated by the tanh function. The dimension of the feature sequence output is consistent with the input. The output of the bidirectional RNN layer is then sent to two FC layers to further extract the acoustic features of the sound event. The first FC layer contains R nodes, each of which has linear activation; the second FC layer consists of N nodes, where the value of N is the number of sound event categories in the corresponding sound dataset; Step 3.3: Construct an acoustic scene classification network to classify acoustic scenes; The feature sequence output in step 3.1 is sent as input to the convolutional neural network. In the convolutional neural network part, a two-layer two-dimensional CNN is used to learn the acoustic features in the feature sequence. The first CNN layer has B 4 The number of filters with a 3×3×2C-dimensional receptive field is also B. 4 ; The step size and padding of the two CNN layers are both 1 to ensure that the feature sequence length T remains unchanged; After each layer of CNN, batch normalization is used to normalize the output and ReLU function is used for activation to keep the feature sequence length T unchanged; the feature sequence output by the second layer of CNN is swapped in dimension and further reshaped before being fed into two FC layers to further extract the acoustic features of the acoustic scene; The first FC layer contains R nodes, each of which has linear activation; the second FC layer consists of M nodes, where the value of M corresponds to the number of sound scenes in the corresponding acoustic dataset; Step 3.4: Build a node representation learning module to learn the node representation in each acoustic scene and then use it as a high-level node to learn the node representation in the driving scene; The node representation learning module uses the GCN network to learn the node representation in each acoustic scene respectively and then uses it as a high-level node to learn the node representation in the driving scene; the GCN network uses two GCN layers, each GCN layer takes the node representation H(l) learned by the previous layer as input, and outputs a new node representation H(l+1) after learning; the input node representation of the first GCN layer is the word embedding of the label, where N represents the number of labels, that is, the number of types of sound events in the sound dataset, and f represents the dimension of the label word embedding; for the last layer, the output is o represents the feature dimension of the output feature sequence, which is the same as the feature sequence dimension output by step 3.2 and step 3.3; the node representation output of multiple acoustic scenes is embedded as the node of the driving scene It is sent to the GCN network again, where M represents the number of acoustic scene types in the driving scene, and F is the embedding dimension of the high-level nodes representing the acoustic scene features; after two layers of GCN, the final output is the feature sequence represented by the node of the driving scene Step 3.5: Transpose the feature sequence represented by the node of the driving scene output in step 3.4; perform matrix multiplication on the feature sequence before transposition and the acoustic feature sequence of sound event detection to obtain a combined first feature sequence; perform corresponding element multiplication on the feature sequence after transposition and the acoustic feature sequence of acoustic scene detection to obtain a combined second feature sequence; Step 3.6: Construct an acoustic scene score output network and a sound event score output network, and implement the final acoustic scene classification and sound event classification tasks based on the scores output by the networks; The two combined feature sequences in step 3.5 are respectively sent to the acoustic scene score output network and the sound event score output network, both of which are implemented by the FC layer; the FC layer of the acoustic scene score output network consists of M nodes with sigmoid activation, each node corresponding to one of the M acoustic scene categories to be detected; The FC layer of the sound event score output network consists of N nodes with sigmoid activation, each node corresponding to one of the N sound event categories to be detected; Using sigmoid activation, multiple classes can be activated at the same time and scores can be output. The cross entropy loss function is used for training and the final classification and detection results are output. The outputs of ASC and SED are both in the continuous range of [0,1]. When the output exceeds the threshold of 0.5, it is determined that the vehicle is in the acoustic scene or the event exists. For the scene determined to be in, the category of the scene is output, and for the sound event determined to exist, the start and end time and duration of the event are output.

2. A detection method for driving scenes and events based on sound perception according to claim 1, Features: The step 1 comprises: Step 1.1: Obtain label information in the traffic scene sound event dataset and perform sound event statistics; Step 1.2: Calculate the relationships between various sound events in the sound event dataset and construct a sound event relationship graph for different driving scenarios; Step 1.3: Incorporate each sound event relationship graph into the driving scene graph as a high-level node. Each node in the driving scene graph represents a driving scene, and each driving scene has a corresponding sound event relationship graph.

3. A method for detecting driving scenes and events based on sound perception according to claim 2, Features: The step 1.1 comprises: The step 1.1.1: reading the label file in the sound event data set, obtaining the label information of the training data set in the sound event data set, including the sound event category in the label file of each audio, and the start and end time of each sound event; Each type of sound event is denoted as L i , i = 0, 1, 2, 3...n, n is the total number of sound events, and the start and end time of each sound event is recorded; Step 1.1.2: Count the number of occurrences of each type of sound event in the sound event data set, the total number of occurrences of all sound events, and the number of times each type of sound event occurs simultaneously in each audio; The situation where two sound events occur simultaneously is recorded as L ij , j = 0, 1, 2, 3 ... n and i ≠ j, L ij Indicates event L i With event L j Occur simultaneously in a piece of audio; according to the sound events with labels in each audio, count the number of occurrences of each type of sound event, recorded as X i , indicating the sound event L i The number of occurrences, X represents the total number of occurrences of various sound events; the number of times two sound events occur at the same time is recorded as X ij Indicates event L i With event L j The number of times it occurs simultaneously in the same audio.

4. The method for detecting driving scenes and events based on sound perception according to claim 3, Features: The step 1.2 comprises: Step 1.2.1: Calculate the probability of various sound events occurring simultaneously using the prior probability formula; Calculate the probability P of various sound events occurring simultaneously ij , according to the number of occurrences of various sound events obtained in step 1.1, X i , and the number of times two sound events occur simultaneously X ij , calculate when the sound event L i When the sound event L occurs j The probability P ij , as shown in the following formula: The probability threshold α, α∈(0,1) is used to filter the sound event relationship, and the sound event relationship weights with probability lower than the threshold are reset to 0, as shown in the following formula: Step 1.2.2: Calculate the occurrence frequency of various sound events in different driving scenarios; Use the number of occurrences X of each event obtained in step 1.1 i , and the total number of times each type of sound event occurs X, calculate the frequency of occurrence of each type of sound event in the scene P i : P i =X i / X Step 1.2.3: Construct the sound event relationship diagram of different driving scenarios; Taking various sound events as nodes, the probability of various sound events occurring simultaneously is P ij The sound event relationship graph is constructed as the directed edges connecting these nodes. The occurrence frequency P of each type of sound event i As part of each node's own properties.

5. The method for detecting driving scenes and events based on sound perception according to claim 1, Features: In step 2, a fast Fourier transform is used on a Hanning window to extract a spectrogram from each C channel of the multi-channel audio; then, the phase and amplitude of the spectrogram are extracted, and the extracted phase and amplitude are combined as separate features into a feature sequence.

6. A method for detecting driving scenes and events based on sound perception according to claim 5, Features: Step 3.1 includes: Step 3.1.1: Create the first two-dimensional CNN layer. The input of this layer is the feature sequence of the preprocessed output in step 2. The CNN layer uses a filter kernel across all channels, allowing the CNN to learn relevant intra-channel features in the time and frequency dimensions. This layer has B 1 A filter with a 3×3×2C-dimensional receptive field, with a step size and padding of 1, is used to ensure that the feature sequence length T remains unchanged. The convolutional layer output is normalized using batch normalization and activated using the ReLU function, and then the maximum pooling along the frequency axis is used to reduce the dimension, thereby keeping the feature sequence length T unchanged. Step 3.1.2: Create the second two-dimensional CNN layer, which has B 2 A filter with a 3×3×2C-dimensional receptive field, with a step size and padding of 1 to ensure that the feature sequence length T remains unchanged; the input of this layer is the feature sequence output by the first CNN layer, which first passes through the convolutional layer, then uses batch normalization to normalize the convolutional layer output and uses the ReLU function for activation, and then reduces the dimension after the maximum pooling along the frequency axis; Step 3.1.3: Create the third two-dimensional CNN layer, which has B 3 The input of this layer is the feature sequence output by the second CNN layer. It first passes through the convolutional layer, then uses batch normalization to normalize the convolutional layer output and uses the ReLU function to activate it. It then uses the maximum pooling along the frequency axis to reduce the dimension.

7. The method for detecting driving scenes and events based on sound perception according to claim 6, Features: Step 3.2 includes: Step 3.2.1: Swap the dimension of the feature sequence output by CNN and further reshape it to a length of 2B 3 The eigenvector of Step 3.2.2: Create two bidirectional RNN layers and feed the feature sequence reshaped in step 3.1.4 into them. Each RNN layer uses a gated recurrent unit (GRU) with Q nodes to extract the temporal context information of the sound event and activates it using the tanh function. The dimension of the feature sequence output is consistent with the input. Step 3.2.3: Create two FC layers and feed the feature sequence of the sound event temporal context information extracted by the bidirectional RNN layer into them for further training; The first FC layer contains R nodes, each with linear activation; The second FC layer consists of N nodes, where the value of N is the number of sound event categories in the corresponding dataset.

8. The method for detecting driving scenes and events based on sound perception according to claim 7, Features: Step 3.3 includes: Step 3.3.1: Create the first two-dimensional CNN layer. The input of this layer is the feature sequence output in step 3.1.

3. The filter kernel across all channels is used to allow CNN to learn relevant intra-channel features in the time and frequency dimensions. This layer has B 4 A filter with a 3×3×2C-dimensional receptive field, a step size and padding of 1 to ensure that the sequence length T remains unchanged; the output is normalized using batch normalization and activated using the ReLU function; Step 3.3.2: Create the second two-dimensional CNN layer, which has B 5 A filter with a 3×3×2C-dimensional receptive field, with a step size and padding of 1 to ensure that the sequence length T remains unchanged; the input of this layer is the feature sequence output by the first CNN layer, which first passes through the convolutional layer, then the output is normalized using batch normalization and activated using the ReLU function; Step 3.3.3: The feature sequence output by the second CNN layer is swapped and further reshaped into a T frame sequence and then fed into two FC layers to further extract the acoustic features of the acoustic scene; Step 3.3.4: Create two FC layers; the first FC layer contains R nodes, each with linear activation; The second FC layer consists of M nodes, where the value of M is the number of sound scenes in the corresponding dataset.

9. The method for detecting driving scenes and events based on sound perception according to claim 8, Features: Step 3.4 includes: Step 3.4.1: Obtain parameters based on the sound event relationship graph structure constructed in step 1, graph G = (V, E), where V contains all N nodes in the sound event relationship network, that is, the N types of sound events to be detected, v i ∈V, E represents the edge between nodes (v i ,v i )∈E; the eigenvector matrix of the node is Where N is the number of nodes and f is the dimension of the feature vector; Step 3.4.2: In order to retain the node's own information while aggregating neighbor information, add self-connections to each node: Among them, A represents the adjacency matrix of graph G, The elements in are weighted real numbers, I N Indicates that each node has added a self-connection, N is the number of nodes, It means that each node has the adjacency matrix after adding the self-connection relationship. Represents the degree matrix after each self-connection is added, the degree matrix D = ∑ j A ij ; Step 3.4.3: Use parameter β to adjust the weight ratio of the node itself and its neighbors. The adjacency matrix after adjusting the weight ratio of itself and its neighbors is as follows: in, Represents the adjacency matrix after adjusting the weight ratio between itself and its neighbors. β is used to allocate the weight ratio between neighbors and itself, β∈[0,1]; Step 3.4.4: Improve the propagation of the GCN network between layers; The improved GCN network propagates between layers according to the following formula: Among them, H(l) represents the input of the l-th layer GCN network, W(l) represents the parameters to be trained, and h() represents the corresponding activation function; Step 3.4.5: Construct the first GCN layer and input the node feature vector matrix H(0) into the GCN network improved by the above step 3.4.4; Among them, H (1) Aggregate the updated node feature vector matrix for the first GCN layer; Step 3.4.6: Construct the second GCN layer and aggregate the updated node feature vector matrix H after the first GCN layer (1) Input the node feature vector matrix after aggregation update, as shown in the following formula: Among them, H (2) Aggregate the updated node feature vector matrix for the second GCN layer; The node feature vector matrix H learned in the last layer (2) The node representation output is used as the acoustic scene; Step 3.4.7: Embed the node representation output of multiple acoustic scenes in step 3.4.6 as the node of the driving scene It is sent to the GCN network again, where M represents the number of acoustic scene types in the driving scene, and F is the embedding dimension of the high-level nodes representing the acoustic scene features; After two layers of GCN, the final output is the feature sequence of the node representation of the driving scene 10. The method for detecting driving scenes and events based on sound perception according to claim 9, Features: Step 3.6 includes: Step 3.6.1: Create an FC layer for acoustic scene detection, and feed the feature sequence combined in step 3.5 into it. The FC layer consists of M nodes with sigmoid activation, each node corresponds to one of the M acoustic scene categories to be detected; using sigmoid activation, multiple classes can be activated at the same time and scores can be output; use the cross entropy loss function for training, and output the final acoustic scene classification result. The acoustic scene detection output is in the continuous range of [0,1]. When the output exceeds the threshold of 0.5, it is determined that the vehicle is in the acoustic scene; Step 3.6.2: Create an FC layer for sound event detection, and feed the first feature sequence M×N combined in step 3.5 into it. The FC layer consists of N nodes with sigmoid activation, each node corresponds to one of the N sound event categories to be detected, and the output of the FC layer is M×N; using sigmoid activation, multiple classes can be activated at the same time and scores can be output; use the cross entropy loss function for training, and output the final sound classification result. The sound event detection output is in the continuous range of [0,1]. When the output exceeds the threshold of 0.5, it is determined that the sound event exists; Step 3.6.3: Output the acoustic scene and sound event category as well as the start and end time and duration of the sound event determined in steps 3.6.1 and 3.6.2.

Images