Driving event judgment method based on sound signals

Through the combination of graph neural network and convolutional recurrent neural network, a sound event relationship diagram is constructed, and co-occurrence and successive relationship information is learned, which solves the problems of detection delay and insufficient real-time in the existing technology, and realizes efficient driving event detection and Quick warning.

CN119066417APending Publication Date: 2024-12-03SHANTOU VOCATIONAL & TECH COLLEGE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411042672.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing technology fails to effectively utilize the co-occurrence and succession relationship between sound events in driving event detection, resulting in delays in detection and inability to promptly warning. Moreover, the model parameters based on deep learning are large and lack of real-time performance, making it difficult to widely use in practice. Scene.

Method used

By combining graph neural network and convolutional recurrent neural network, by constructing a sound event relationship diagram, we can obtain the co-occurrence and successive relationship information of sound events, and design the acoustic feature learning module and node representation learning module to learn two relationship information, and use In sound event detection and early warning.

Benefits of technology

It improves the classification and detection capabilities of polyphonic sound events, realizes high real-time early warning, and can respond quickly before the sound event detection results appear, reducing risks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066417A_ABST
    Figure CN119066417A_ABST
Patent Text Reader

Abstract

The invention provides a driving event judgment method based on sound signals, and relates to the technical field of automobile intelligent cabins. According to the method, a sound event association window is used for each sound event, co-occurrence information and successive information of the sound events are obtained respectively, and a sound event co-occurrence relation graph and a sound event successive relation graph are constructed respectively according to the collinear relation and the successive relation of the events; audio data in the sound event data set are preprocessed, and a feature sequence is extracted; and then carrying out function fusion on the graph neural network and the convolutional recurrent neural network, designing a sound event detection model and carrying out sound event detection, and designing a volume-based rapid early warning module to carry out sound event early warning. According to the method, faster early warning and response can be carried out before a sound event detection result is obtained, risks are reduced, and time is provided for analysis and detection of specific crisis events.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of automobile intelligent cockpits, and in particular to a method for determining a driving event based on sound signals. Background Art

[0002] With the rapid development of the automotive industry towards intelligent networking, various types of sensor technologies are applied to smart cars, and the perception ability of cars is becoming stronger and stronger. In addition to visual technology, sound signals can also provide a lot of useful information, and have the advantages of not requiring light, and not being disturbed by rainy days and dark days. Audio monitoring equipment using sound event detection technology will be widely used in smart cars in the future due to its low cost, small size, easy and fast installation, strong reliability, not easy to damage, and simple maintenance.

[0003] In recent years, the technical framework for sound event classification and detection is mainly based on the deep learning route, using convolutional neural networks (CNN) and recurrent neural networks (RNN) to build detection models. The features in the audio signal are extracted and classified as the basis for sound event analysis and detection, and the detection results are directly obtained based on this. However, the model does not take into account the connection between sound events in the scene to help the model make predictions. At the same time, the relationship information between each sound event and other sound events that occur successively is not effectively utilized. Therefore, there is still a certain gap compared with humans in the detection of polyphonic sound events, especially when there are multiple types of sound events in the audio. And because the existing deep learning-based models have a large number of parameters and lack of real-time performance, they have not been widely used in actual scenarios, especially in driving scenarios with high real-time requirements.

[0004] When the vehicle is in normal condition and driving normally, the vehicle itself and the surrounding environment usually do not have too much noise. When a loud noise suddenly appears, it is usually accompanied by some emergencies, such as the violent collision sound of a car accident, people's screams, natural disasters such as mudslides and landslides, and the vehicle itself will also make a loud noise when it has problems such as tire blowouts. Therefore, the sudden loud noise during driving usually indicates a possible crisis. The instantaneous detection time of the volume is very short, and the detection can be completed within tens of milliseconds. It has high real-time performance and can provide rapid warnings.

[0005] Currently, there is no technical solution that can comprehensively extract the co-occurrence relationship and successive relationship between each sound event, and use the two relationships to detect sound events, while providing rapid warnings based on volume, reminding the driver in a very short time or making corresponding adjustments to the vehicle in autonomous driving.

[0006] Keisuke Imoto and Seisuke Kyochi pointed out in their paper that the types of sound events that occur in a single scene are limited, and some sound events occur at the same time. Therefore, it is hoped that sound events can be efficiently modeled using limited sound data. The occurrence of sound events is represented as a graph, whose nodes represent the frequency of event occurrence and whose edges represent co-occurring sound events. This graph is used for sound event modeling, and graph Laplacian regularization is introduced into the objective function of the neural network. This method takes into account the simultaneous occurrence of sound events in each audio segment. Helin Wang and Yuexian Zou considered the probability of co-occurrence between sound events in the existing audio annotation dataset in their paper. To solve this problem, the label dependency is modeled in the paper through a graph-based method, where each node of the graph represents a label. The structural information of the graph is represented by constructing an adjacency matrix by mining the statistical relationship between the labels, and the node representation is learned by using a graph convolutional network (GCN) to implicitly model the label dependency. The generated node representation is then applied to the acoustic representation for classification.

[0007] Existing technologies almost all use neural network models such as CNN, RNN, CRNN, etc. to detect sound events based on the independent features of each sound event in the audio signal. They do not use the relationship between sound events to help the model make predictions, nor do they consider the temporal relationship of the sound events. In addition, when encountering dangerous situations, the detection time required by existing technologies may delay the opportunity and fail to respond in time to avoid accidents.

[0008] When constructing a sound event relationship graph, existing methods only consider sound events that occur at the same time. They use the probability of two sound events occurring at the same time to construct an undirected graph or use the conditional probability of two sound events occurring at the same time to construct a directed graph. Neither method takes into account the situation where sound events occur one after another. At the same time, due to its harsh conditions, the graph structure constructed using the co-occurrence method basically has many nodes but few edges. That is, the graph constructed by this method is relatively sparse, and the transmission efficiency is low when using a graph network to learn a sparse graph. Shallow graph neural networks cannot obtain enough relationship information, and deepening the graph network will increase the amount of computation while facing the risk of over-smoothing. In addition, many related sound events also have a certain temporal sequence. This relationship between the occurrence of events one after another is also very important for the inference of sound events.

[0009] The existing technologies do not take into account the situations where multiple events occur simultaneously and successively in a scene, nor do they construct sound event relationship graphs for the two relationships of simultaneous and consecutive sound events, and use them as effective information for event analysis. Currently, there is no sound event recognition technology that uses graph neural networks to learn sound event relationships and sound event succession relationships respectively, detects the timing relationship and start and end time of sound events, and uses sudden loud noises as warning information during driving. Summary of the invention

[0010] The technical problem to be solved by the present invention is to provide a method for judging a traffic event based on sound signals in view of the above-mentioned deficiencies in the prior art, so as to judge and warn of traffic events.

[0011] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for judging a driving event based on a sound signal, comprising the following steps:

[0012] Step 1: Construct a sound event relationship graph, including the following steps:

[0013] Step 1.1: Obtain and process the label information of the sound event;

[0014] Read the label file in the sound event dataset to obtain the label information of the training dataset, including the sound event category in the label file of each audio, and the start and end time of each sound event;

[0015] Each type of sound event is denoted as L i , i = 0, 1, 2, 3...n, n is the total number of sound event categories, and the start and end time of each sound event is recorded, where each sound event is recorded as M is the total number of sound events, and the start time of each sound event is expressed as The end time is expressed as

[0016] Step 1.2: Design the sound event association window;

[0017] A sound event association window of length T is used at the end time of each sound event in the audio in the sound event dataset to obtain other sound events that occur successively after each sound event, thereby achieving detection performance of the successive occurrence of sound events;

[0018] Step 1.3: Calculate the sound event relationship and construct the sound event co-occurrence relationship graph and the succession relationship graph, including the following steps:

[0019] Step 1.3.1: Obtain the sound event information of the co-occurrence relationship;

[0020] Each sound event obtained in step 1.1 Start and end time and Get every sound event The co-occurrence range Other sound events within, when a sound event L i m When there are other sound events within the co-occurrence range of and i≠j; the number of times the sound event co-occurs is recorded as Indicates sound event L i With sound events L j The number of simultaneous occurrences, i.e., the number of co-occurrences;

[0021] Step 1.3.2: Obtaining sound event information of successive association relationships;

[0022] Each sound event obtained in step 1.1 Start and end time and Combined with the length T of the sound event association window, obtain each sound event L i m The sound event correlation window range Other sound events within a sound event If there are other sound events within the sound event association window, it is considered as a sequential relationship, recorded as The number of times two consecutive sound events occur in succession is recorded as Indicates sound event L i With sound events L j The number of occurrences within the same sound event window;

[0023] Step 1.3.3: Obtain the number of occurrences of each type of sound event in the sound event dataset;

[0024] Count the number of occurrences of various sound events in the sound event dataset, denoted as X i , indicating the sound event L i Number of occurrences;

[0025] Step 1.3.4: Calculate the probability of simultaneous and sequential occurrence of various sound events in the sound event data set using the prior probability formula;

[0026] According to the number of occurrences of various sound events in the sound event data set X i and the number of times the audio events co-occur Calculate when the sound event L i When the sound event L occurs j Probability As shown in the following formula:

[0027]

[0028] in, For the sound event L i When the sound event L occurs j The probability of is the number of co-occurrences of audio events, X i is the number of occurrences of various sound events in the sound event dataset;

[0029] According to the number of occurrences of various events X i and the number of times the event occurred Calculate the probability of sound events happening one after another As shown in the following formula:

[0030]

[0031] The probability thresholds α1, α2, α1, α2∈(0,1) are used to filter the sound events, and the sound events with probabilities lower than the probability thresholds are used as the relationship weights between nodes, and the relationship weights are reset to 0, as shown in the following formula:

[0032]

[0033] Finally, various sound events are used as nodes and the calculated probabilities are and The directed edges connecting these nodes are used as the sound event co-occurrence relationship graph and the sound event succession relationship graph respectively, so as to complete the construction of the sound event co-occurrence relationship graph and the sound event succession relationship graph;

[0034] Step 2: Preprocess the audio data in the sound event dataset and extract the feature sequence, including the following steps:

[0035] A 512-point fast Fourier transform is used to extract a spectrogram from each C channel of the multi-channel audio on a Hanning window with a length of 512 and an overlap of 50%. The phase and amplitude of the spectrogram are then extracted and used as separate features, where the amplitude value is also used to determine the volume in the early warning technology.

[0036] Step 2.1: Set the audio sampling frequency and process the audio data into frames;

[0037] Step 2.2: Select a window function to perform a frame-by-frame windowing operation on the framed audio data obtained in step 1;

[0038] Step 2.3: Obtain the spectrum and extract the phase and amplitude;

[0039] Step 2.3.1: Convert the framed and windowed audio data from the time domain to the frequency domain through fast Fourier transform to obtain the frequency domain signal of the audio data

[0040] Step 2.3.2: Perform a modulo operation on the frequency domain signal obtained in step 2.3.1 to obtain the amplitude spectrum of the audio data;

[0041] Step 2.3.3: Perform a phase angle operation on the frequency domain signal obtained in step 2.3.1 to obtain the phase spectrum of the audio data

[0042] Step 2.3.4: Combine the amplitude spectrum and phase spectrum obtained in step 2.3.2 and step 2.3.3 to obtain a characteristic sequence;

[0043] Step 3: Design a sound event detection model and perform sound event detection, including the following steps:

[0044] Step 3.1: Design an acoustic feature learning module to perform acoustic feature learning; the acoustic feature learning module includes a convolutional neural network, a bidirectional RNN layer and two FC layers; the convolutional neural network uses a three-layer two-dimensional CNN to learn the acoustic features in the feature sequence and inputs them into the bidirectional RNN layer; the bidirectional RNN layer is used to learn the temporal context information from the CNN output and input it into the two FC layers; the two FC layers further extract the acoustic features of the sound event;

[0045] The feature sequence obtained by preprocessing in step 2 is sent as input to the convolutional neural network; in the convolutional neural network part, a three-layer two-dimensional CNN is used to learn the acoustic features of the feature sequence; the first CNN layer has B1 filters with a 3×3×2C-dimensional receptive field, B1 is 256, C is 4, and the number of filters in the next two CNN layers is 128 and 64 respectively; the step size and padding of the three CNN layers are both 1 to ensure that the feature sequence length T remains unchanged; after each layer of CNN, the output is standardized using batch normalization and activated using the ReLU function, and then the maximum pooling along the frequency axis is used to reduce the dimension, thereby keeping the feature sequence length T unchanged; the three CNN layers use filter kernels that span all channels, allowing CNN to learn relevant intra-channel features in the time and frequency dimensions; the filter output of the last CNN layer is T×2×B3, that is, 512×2×64, where the frequency dimension is reduced to 2 due to the effect of the maximum pooling layer; after each of the three CNN layers, the maximum pooling on the frequency axis is respectively 8, 4, 4; the feature sequence 512×2×64 output by the three CNN layers is exchanged in dimension and further reshaped into a T frame sequence of feature vectors of length 2B3, i.e., 512×128, and sent to the bidirectional RNN layer, which is used to learn temporal context information from CNN output; a gated recurrent unit of Q nodes is used in each RNN layer, and the gated recurrent unit is used to extract the temporal context information of sound events to obtain the timing information of various sound events in the audio, the value of Q is 128, and the tanh function is used for activation, and the dimension of the feature sequence output is finally kept consistent with the input; then the feature sequence output by the bidirectional RNN layer is sent to two FC layers to further extract the acoustic features of the sound events; the first FC layer contains R nodes, R is 128, and each node has linear activation; the second FC layer consists of N nodes, and the value of N is the number of sound event categories in the corresponding data set; the feature sequence of the acoustic features outputted finally is 512×N; the following steps are included:

[0046] Step 3.1.1: Create the first two-dimensional CNN layer. The input of this layer is the feature sequence 512×256×8 output by step 2 preprocessing. Use the filter kernel across all channels to allow CNN to learn relevant intra-channel features in the time and frequency dimensions. This layer has B1 filters with 3×3×2C-dimensional receptive fields, B1 is 256, C is 4, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; after passing through the CNN layer, the feature sequence becomes 512×256×256; use batch normalization to standardize the CNN layer output and activate it with the ReLU function, then use the maximum pooling along the frequency axis to reduce the dimension to 512×32×256, thereby keeping the feature sequence length T unchanged, and the maximum pooling on the frequency is 1×8;

[0047] Step 3.1.2: Create a second two-dimensional CNN layer, which has B2 filters with 3×3×8 dimensional receptive fields, B2 is 128, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; the input of this layer is the feature sequence 512×32×256 output by the first CNN layer, which first passes through the convolution layer. After convolution, the feature sequence dimension becomes 512×32×128, and then the output is standardized using batch normalization and activated using the ReLU function. After that, the maximum pooling along the frequency axis is reduced to 512×8×128, and the maximum pooling on the frequency is 1×4;

[0048] Step 3.1.3: Create the third two-dimensional CNN layer, which has B3 filters with 3×3×8 dimensional receptive fields, B3 is 64, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; the input of this layer is the feature sequence 512×8×128 output by the second CNN layer. First, it passes through the convolution layer. After the convolution, the feature sequence dimension becomes 512×8×64, and then the output is standardized using batch normalization and activated using the ReLU function. After that, the maximum pooling along the frequency axis is used to reduce the dimension to 512×2×64, and the maximum pooling on the frequency is 1×4;

[0049] Step 3.1.4: The feature sequence 512×2×64 output by the three-layer CNN layer is swapped into 2×512×64 and further reshaped into a T-frame sequence of feature vectors of length 2B3, i.e. 512×128;

[0050] Step 3.1.5: Create two bidirectional RNN layers and feed the feature sequence reshaped in step 3.1.4 into them to extract the temporal context information of the sound event. Each RNN layer uses a gated recurrent unit with Q nodes, Q is 128, and is activated by the tanh function. The dimension of the final output feature sequence is consistent with the input, that is, 512×128;

[0051] Step 3.1.6: Create two FC layers and feed the feature sequence of the sound event temporal context information extracted by the RNN layer into them for further training; the first FC layer contains R nodes, R is 128, and each node has linear activation. The second FC layer consists of N nodes, and the value of N is the number of sound event categories in the corresponding data set. The feature sequence of the final output acoustic features is 512×N;

[0052] Step 3.2: Design a node representation learning module to learn the node representation of two types of relationship information, including the following steps:

[0053] The node representation learning module uses the GCN network to learn the node representations of co-occurrence and successive relationship information respectively, where and They represent the node representation of co-occurrence relationship and the node representation of succession relationship respectively; the GCN network uses two GCN layers, each GCN layer represents the node learned by the previous layer or As input, output new node representation after learning or Input node representation of the first GCN layer is the word embedding of the label, where N represents the number of labels, that is, the number of types of sound events in the dataset, and f represents the dimension of the label word embedding; for the last layer, the output is Where o is equal to the dimension of the acoustic features output by the acoustic feature learning module;

[0054] Step 3.2.1: Obtain the sound event co-occurrence relationship graph and the sound event succession relationship graph constructed in step 1 as G = (V, E), where V contains all N nodes in the network, i.e., the N sound events to be detected, v i ∈V, E represents the edge between nodes (v i ,v i )∈E, the eigenvector matrix of the node is Where N is the number of nodes and f is the dimension of the feature vector;

[0055] Step 3.2.2: In order to retain the node's own information while aggregating neighbor information, add self-connections to each node:

[0056]

[0057] Where A represents the adjacency matrix, Element A in ij is a real weight; I N Indicates that each node has a self-connection added. It means that each node has the adjacency matrix after adding the self-connection relationship. Represents the degree matrix after each self-connection is added, the degree matrix D = ∑ j A ij ;

[0058] Step 3.2.3: Use parameter β to adjust the weight ratio between the node itself and its neighbors:

[0059]

[0060] in, Represents the adjacency matrix after adjusting the weight ratio between itself and its neighbors. β is used to allocate the weight ratio between neighbors and itself, β∈[0,1];

[0061] Step 3.2.4: The propagation of the improved GCN network between layers follows the following formula:

[0062]

[0063] Among them, H (l) Represents the input of the l-th layer GCN network, W (l) It represents the parameters of the l-th layer GCN network to be trained, and h() represents the corresponding activation function.

[0064] Step 3.2.5: Construct the first GCN layer and transform the node feature vector matrix Input the GCN network improved by step 3.2.4 above to learn the node representation As shown in the following formula:

[0065]

[0066] Step 3.2.6: Construct the second GCN layer and aggregate the updated node feature vector matrix after the first GCN layer Input it and learn the node representation As shown in the following formula:

[0067]

[0068] The node representation learned in the last layer As the output of the node representation learning module, the feature sequence of the node representation outputted is N×512, where and Representing the node representation of co-occurrence relationship and the node representation of succession relationship respectively;

[0069] Step 3.3: Transpose the feature sequences N×512 represented by the nodes output by the two types of relationship information learned by GCN to 512×N, and then fuse them in the following two ways to obtain the combined feature sequence:

[0070] Method 1: The two obtained nodes represent the feature sequence of learning output and Perform a corresponding element multiplication, and then perform a corresponding element multiplication with the feature sequence output by acoustic feature learning, combine the two node representation outputs and the acoustic feature output, and obtain the combined feature sequence 512×N;

[0071] Method 2: Two types of nodes represent the output feature sequence and Perform weighted combination and use parameter γ to adjust the weight ratio of the feature sequences of the two nodes to represent the output. The combination is as follows:

[0072]

[0073] in, It represents the feature sequence of nodes with two kinds of relationship information after weighted combination. The parameter γ is used to adjust the weight ratio of the two. Then the node representation after weighted combination is Multiply the corresponding elements with the feature sequence output by acoustic feature learning, combine the two node representation outputs with the acoustic feature output, and obtain the combined feature sequence 512×N;

[0074] Step 3.4: Design a sound event classification network to classify sound events, including the following steps:

[0075] The combined feature sequence is sent to a sound event classification network for sound event classification; the sound event classification network is composed of two FC layers; the first FC layer contains R nodes, each of which has linear activation; the second FC layer is composed of N nodes with sigmoid activation, each of which corresponds to one of the N sound event categories to be detected, and multiple classes can be activated simultaneously using sigmoid activation, and scores are output, and the cross entropy loss function is used for training, and the final sound classification and detection results are output. The output of the sound event detection is in the continuous range of [0,1]. When the output exceeds the threshold value of 0.5, it is judged that the sound event exists, and the start and end time and duration of the sound event that is judged to exist are output;

[0076] Step 4: Design a volume-based rapid warning module for sound event warning, including the following steps:

[0077] Step 4.1: Determine the amplitude of the sound wave and obtain the volume at the current moment using the amplitude value of the audio signal obtained in step 2.3.2. The amplitude value is recorded as V t , the value of t is the current moment;

[0078] Step 4.2: Determine the volume threshold based on the sound event detection model and statistical analysis;

[0079] 1) Use audio data to train the sound event detection model;

[0080] 2) Use the trained sound event detection model to perform sound event detection on the audio data;

[0081] 3) According to the detection results of the sound event detection model, the volume amplitude values ​​of different sound event categories are counted and risk assessment and classification are performed according to the sound event categories. Subsequently, a large amount of data is statistically analyzed to select the optimal classification threshold Y;

[0082] Step 4.3: Based on the acquired amplitude value and volume threshold, realize the warning based on the high volume;

[0083] Use the volume threshold Y to perform two classifications. When the amplitude value V of the audio signal t When ≤Y, it is considered as risk-free situation, V t A quick risk warning will be issued when ≥Y.

[0084] The beneficial effects of the above technical solution are: a driving event judgment method based on sound signals provided by the present invention integrates the functions of graph neural network and convolutional recurrent neural network, uses sound event association window for each sound event, obtains the co-occurrence information and successive information of sound events respectively, and composes them according to the collinear relationship and successive relationship of events respectively. The composition mode and the two relationship information are independently learned without interfering with each other. The model can effectively obtain two different relationship information, which is more comprehensive than the traditional co-occurrence method. At the same time, it also effectively alleviates the sparse graph problem faced by the traditional co-occurrence method, so that the sound event detection task can obtain more prior knowledge. Effectively improve the ability of classification and detection of polyphonic sound events. At the same time, the application also regards the sudden appearance of loud noise during driving as possible crisis information, and uses the real-time detection of volume to perform high-real-time early warning technology during driving. This technology has high real-time performance and almost no increase in computational complexity. It can perform faster early warning and response before the sound event detection results come out, reduce risks, and provide time for the analysis and detection of specific crisis events. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A flow chart of a method for determining a driving event based on a sound signal provided by an embodiment of the present invention;

[0086] Figure 2 A schematic diagram of a sound event association window provided by an embodiment of the present invention;

[0087] Figure 3 A sound event relationship graph provided by an embodiment of the present invention, wherein (a) is a sound event co-occurrence relationship graph, and (b) is a sound event succession relationship graph;

[0088] Figure 4 A flowchart of audio data preprocessing provided by an embodiment of the present invention;

[0089] Figure 5 A model structure diagram of sound event detection provided by an embodiment of the present invention;

[0090] Figure 6 A schematic diagram of risk warning based on volume level provided by an embodiment of the present invention;

[0091] Figure 7 A flowchart of the combined use of the sound warning technology and the vehicle-mounted intelligent system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0092] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0093] In this embodiment, a method for determining a driving event based on a sound signal is provided. Figure 1 As shown, the following steps are included:

[0094] Step 1: Construct a sound event relationship graph, including the following steps:

[0095] Step 1.1: Obtain and process the label information of the sound event;

[0096] Read the label file in the sound event dataset to obtain the label information of the training dataset, including the sound event category in the label file of each audio, and the start and end time of each sound event;

[0097] Each type of sound event is denoted as L i , i = 0, 1, 2, 3...n, n is the total number of sound event categories, and the start and end time of each sound event is recorded, where each sound event is recorded as M is the total number of sound events, and the start time of each sound event is expressed as The end time is expressed as

[0098] Step 1.2: Design the sound event association window;

[0099] The sound event association window technology proposed in the present invention is similar to the way people judge the sequential relationship of events, that is, if another event occurs within a certain period of time after one event occurs, and when it occurs repeatedly, it can be considered that the two events have a certain correlation.

[0100] A sound event association window of length T is used at the end time of each sound event in the audio in the sound event dataset to obtain other sound events that occur successively after each sound event, thereby achieving the detection performance of the sound events that occur successively; Figure 2 shown.

[0101] Figure 2 The box in represents a sound event association window of length T added to the end time of each sound event. The sound event co-occurrence is used in combination with the sound event association window to obtain more comprehensive sound event relationship information. After using the sound event association window, the relationship between the voice and the footsteps that could not be obtained by co-occurrence was successfully obtained. The correlation between the voice and the footsteps that could not be obtained by the co-occurrence method is also more in line with our cognition and the actual situation in life.

[0102] Step 1.3: Calculate the sound event relationship and construct the sound event co-occurrence relationship graph and the succession relationship graph, including the following steps:

[0103] Step 1.3.1: Obtain the sound event information of the co-occurrence relationship;

[0104] Each sound event obtained in step 1.1 Start and end time and Get every sound event The co-occurrence range Other sound events within a sound event When there are other sound events within the co-occurrence range of and i≠j; the number of times the sound event co-occurs is recorded as Indicates sound event L i With sound events L j The number of simultaneous occurrences, i.e., the number of co-occurrences;

[0105] Step 1.3.2: Obtaining sound event information of successive association relationships;

[0106] Each sound event L obtained in step 1.1 i m Start and end time and Combined with the length T of the sound event association window, obtain each sound event The sound event correlation window range Other sound events within a sound event If there are other sound events within the sound event association window, it is considered as a sequential relationship, recorded as The number of times two consecutive sound events occur in succession is recorded as Indicates sound event L i With sound events L j The number of occurrences within the same sound event window;

[0107] Step 1.3.3: Obtain the number of occurrences of each type of sound event in the sound event dataset;

[0108] Count the number of occurrences of various sound events in the sound event dataset, denoted as X i , indicating the sound event L i Number of occurrences;

[0109] Step 1.3.4: Calculate the probability of simultaneous and sequential occurrence of various sound events in the sound event data set using the prior probability formula;

[0110] According to the number of occurrences of various sound events in the sound event data set X i and the number of times the audio events co-occur Calculate when the sound event L i When the sound event L occurs j Probability As shown in the following formula:

[0111]

[0112] It should be noted that and They are not necessarily equivalent and their meanings are also different.

[0113] in, For the sound event L i When the sound event L occurs j The probability of is the number of co-occurrences of audio events, X i is the number of occurrences of various sound events in the sound event dataset;

[0114] Similarly, according to the number of occurrences of various events X i and the number of times the event occurred Calculate the probability of sound events happening one after another As shown in the following formula:

[0115]

[0116] Similarly, and They are not necessarily equal, and the meanings they express are also different.

[0117] Since there are some differences in the probabilities of various events between the training data and the data obtained in actual use, in order to avoid some interference relationships with very low probabilities calculated using the training data from having an adverse effect on the model test, and to avoid the model from overfitting the graph structure generated by the training set during training, the probability thresholds α1, α2, α1, α2∈(0,1) are used to filter the sound events, and the sound events with probabilities lower than the probability thresholds are used as the relationship weights between the nodes, and the relationship weights are reset to 0, as shown in the following formula:

[0118]

[0119] This operation can effectively reduce possible interference information.

[0120] Finally, various sound events are used as nodes and the calculated probabilities are and The directed edges connecting these nodes are used as the sound event co-occurrence relationship graph and the sound event succession relationship graph respectively, so as to complete the construction of the sound event co-occurrence relationship graph and the sound event succession relationship graph;

[0121] The sound event co-occurrence relationship graph and the sound event succession relationship graph are constructed separately. Some examples of the constructed graphs are as follows: Figure 3 As shown, Figure 3 (a) is the edge in the sound event co-occurrence relationship graph, which represents the co-occurrence relationship of sound events. Figure 3 (b) is a sound event succession relationship diagram representing the succession relationship of sound events, and the numbers all represent probabilities. This embodiment uses some sounds picked up during car driving as an example to illustrate the constructed sound event co-occurrence relationship diagram and sound event succession relationship diagram, wherein the sound data includes talking sounds, footsteps, the sound of a driving vehicle, honking sounds, the sound of a violent collision in a car accident, and the sound of brakes.

[0122] Figure 3 In the sound event co-occurrence graph, there is an edge pointing from footsteps to speaking, and there is also an edge pointing to footsteps, and each edge has a number, which means that when footsteps appear, there is a 42% probability that speaking will appear at the same time, and when speaking appears, there is a 47% probability that footsteps will appear at the same time.

[0123] Figure 3 In the sound event sequence relationship graph, there is an edge pointing from the footsteps to the speaking sound and an edge pointing from the violent collision sound of a car accident to it. There is also an edge pointing to the footsteps, and each edge has a number, which means that when footsteps appear, there is a 72% probability that speaking sounds will appear successively, when the violent collision sound of a car accident appears, there is an 86% probability that speaking sounds will appear successively, and when speaking sounds appear, there is an 83% probability that footsteps will appear successively.

[0124] The two constructed graph structures can be used as two independent inputs, and the graph neural network can be used to learn the co-occurrence relationship and succession relationship in sound events respectively, so as to obtain more comprehensive event relationship information while the two sound event relationships do not interfere with each other.

[0125] Step 2: Preprocess the audio data in the sound event dataset and extract feature sequences, such as Figure 4 As shown, the following steps are included:

[0126] On a Hanning window with a length of 512 and an overlap of 50%, a 512-point fast Fourier transform (FFT) is used to extract a spectrogram from each C channel of the multi-channel audio; then the phase and amplitude of the spectrogram are extracted, and the extracted phase and amplitude are used as separate features, where the amplitude value is also used to determine the volume in the early warning technology; in order to improve the efficiency of audio signal processing and simplify model design, only half of the positive frequencies are used, and the zeroth bit is not used. The choice of this method is based on the symmetry of the negative frequency part and the positive frequency part. The two contain the same information but opposite signs. Therefore, in this way, the amount of calculation and storage requirements can be effectively reduced, the calculation efficiency can be improved, and the information of the negative frequency part can be retained. Figure 4 The output feature vector is a feature sequence of T frames, with an overall size of T×256×2C, where 2C is composed of C amplitude and C phase components.

[0127] Step 2.1: Set the audio sampling frequency to 44.1KHz, divide the audio data into frames, set the frame length T to 512 sampling points, and the frame shift J to half of the frame length, that is, 256 sampling points. The speech vector of the collected audio at frame k is shown in the following formula:

[0128] x k (m) = x[kJ:KJ+M-1],1 <m<M,k∈[0,K]

[0129] Where k is the frame index value, K is the total number of audio frames, and M is the total length of the signal. Represents the speech vector of the collected audio at frame k;

[0130] Step 2.2: Select a window function to perform a frame-by-frame windowing operation on the framed audio data obtained in step 1, including the following steps:

[0131] Step 2.2.1: Select Hanning window as the window function;

[0132] The Hanning window function is shown in the following formula:

[0133]

[0134] Among them, w(l win ) represents the Hanning window function, a is the coefficient, L win is the window length; if a is set to 0.5, the Hanning window will be obtained, and the formula can be simplified to

[0135]

[0136] Step 2.2.2: Perform a windowing operation frame by frame on the framed audio data obtained in step 1, as shown in the following formula:

[0137] y k (m) = x k (m)w(l win )

[0138] Among them, x k (m) represents the audio data to be windowed, Represents the kth frame data of the windowed audio data in the time domain;

[0139] Step 2.3: Obtain the spectrum and extract the phase and amplitude, including the following steps:

[0140] Step 2.3.1: Convert the framed and windowed audio data from the time domain to the frequency domain through fast Fourier transform to obtain the frequency domain signal of the audio data, as shown in the following formula:

[0141]

[0142] in, It is a complex number, representing the data after FFT transformation, and the real part represents the lth frequency band in the kth frame. fft The amplitude of the frequency component is used for volume judgment in early warning technology. The imaginary part represents the lth frequency component in the mth frequency band of the kth frame. fft The phase of the frequency component, L fft is the length of the fast Fourier transform;

[0143] Step 2.3.2: Perform a modulo operation on the frequency domain signal obtained in step 2.3.1 to obtain the amplitude spectrum of the audio data, as shown in the following formula:

[0144] U k (l fft )=|S k (l fft )|

[0145] Among them, U k (l fft ) represents the lth frequency band in the kth frame fft The amplitude component of the frequency component;

[0146] Step 2.3.3: Perform a phase angle operation on the frequency domain signal obtained in step 2.3.1 to obtain the phase spectrum of the audio data, as shown in the following formula:

[0147] θ k (l fft )=argtan(S k (l fft ))

[0148] Among them, θ k (l fft) is the lth frequency band of the kth frame of audio data fft The phase of the frequency components;

[0149] Step 2.3.4: Combine the amplitude spectrum and phase spectrum obtained in step 2.3.2 and step 2.3.3 into a feature sequence with a shape of 512×256×2C; this embodiment uses four microphones to receive the audio signal outside the car, so C is 4, where the 2C size consists of C amplitude and C phase components. The final shape of the feature sequence is 512×256×8.

[0150] Step 3: Design a sound event detection model and perform sound event detection, including the following steps:

[0151] Step 3.1: Design an acoustic feature learning module to perform acoustic feature learning; the acoustic feature learning module includes a convolutional neural network, a bidirectional RNN layer and two FC layers; the convolutional neural network uses a three-layer two-dimensional CNN to learn the acoustic features in the feature sequence and inputs them into the bidirectional RNN layer; the bidirectional RNN layer is used to learn the temporal context information from the CNN output and input it into the two FC layers; the two FC layers further extract the acoustic features of the sound event;

[0152] The process is as follows: Figure 5As shown in the figure, the feature sequence obtained by preprocessing in step 2 is sent as input to the convolutional neural network; in the convolutional neural network part, a three-layer two-dimensional CNN is used to learn the acoustic features of the feature sequence; the first CNN layer has B1 filters with a 3×3×2C-dimensional receptive field, B1 is 256, C is 4, and the number of filters in the next two CNN layers is 128 and 64 respectively; the step size and padding of the three CNN layers are both 1 to ensure that the feature sequence length T remains unchanged; after each layer of CNN, the output is standardized using batch normalization and activated using the ReLU function, and then the maximum pooling along the frequency axis is used to reduce the dimension, thereby keeping the feature sequence length T unchanged; the three CNN layers use filter kernels that span all channels, allowing CNN to learn relevant intra-channel features in the time and frequency dimensions; the filter output of the last CNN layer is T×2×B3, that is, 512×2×64, where the frequency dimension is reduced to 2, which is the effect of the maximum pooling layer; after each of the three CNN layers, the maximum pooling on the frequency The feature sequences of the three CNN layers are exchanged in dimension and further reshaped into a T-frame sequence of feature vectors of length 2B3, i.e., 512×128, and sent to the bidirectional RNN layer, which is used to learn temporal context information from the CNN output; in each RNN layer, a gated recurrent unit (GRU) with Q nodes is used to extract the temporal context information of the sound event to obtain the timing information of various sound events in the audio, the value of Q is 128, and the tanh function is used for activation, and the dimension of the feature sequence output is consistent with the input; then the feature sequence output by the bidirectional RNN layer is sent to two FC layers to further extract the acoustic features of the sound event; the first FC layer contains R nodes, R is 128, and each node has linear activation; the second FC layer consists of N nodes, and the value of N is the number of sound event categories in the corresponding data set; the feature sequence of the acoustic features output is 512×N. It includes the following steps:

[0153] Step 3.1.1: Create the first two-dimensional CNN layer. The input of this layer is the feature sequence 512×256×8 output by step 2 preprocessing. Use the filter kernel across all channels to allow CNN to learn relevant intra-channel features in the time and frequency dimensions. This layer has B1 filters with 3×3×2C-dimensional receptive fields, B1 is 256, C is 4, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; after passing through the CNN layer, the feature sequence becomes 512×256×256; use batch normalization to standardize the CNN layer output and activate it with the ReLU function, then use the maximum pooling along the frequency axis to reduce the dimension to 512×32×256, thereby keeping the feature sequence length T unchanged, and the maximum pooling on the frequency is 1×8;

[0154] Step 3.1.2: Create a second two-dimensional CNN layer, which has B2 filters with 3×3×8 dimensional receptive fields, B2 is 128, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; the input of this layer is the feature sequence 512×32×256 output by the first CNN layer, which first passes through the convolution layer. After convolution, the feature sequence dimension becomes 512×32×128, and then the output is standardized using batch normalization and activated using the ReLU function. After that, the maximum pooling along the frequency axis is reduced to 512×8×128, and the maximum pooling on the frequency is 1×4;

[0155] Step 3.1.3: Create the third two-dimensional CNN layer, which has B3 filters with 3×3×8 dimensional receptive fields, B3 is 64, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged. The input of this layer is the feature sequence 512×8×128 output by the second CNN layer. It first passes through the convolution layer, and the feature sequence dimension becomes 512×8×64 after convolution. Then, the output is standardized using batch normalization and activated using the ReLU function. After that, the maximum pooling along the frequency axis is used to reduce the dimension to 512×2×64, and the maximum pooling on the frequency is 1×4;

[0156] Step 3.1.4: The feature sequence 512×2×64 output by the three-layer CNN layer is swapped into 2×512×64 and further reshaped into a T-frame sequence of feature vectors of length 2B3, i.e., 512×128.

[0157] Step 3.1.5: Create two bidirectional RNN layers and feed the feature sequence reshaped in step 3.1.4 into them to extract the temporal context information of the sound event. Each RNN layer uses a gated recurrent unit (GRU) with Q nodes, Q is 128, and is activated by the tanh function. The dimension of the final output feature sequence is consistent with the input, that is, 512×128;

[0158] Step 3.1.6: Create two FC layers and feed the feature sequence of the sound event temporal context information extracted by the RNN layer into them for further training. The first FC layer contains R nodes, R is 128, and each node has linear activation. The second FC layer consists of N nodes, where the value of N is the number of sound event categories in the corresponding dataset. The feature sequence of the final output acoustic features is 512×N;

[0159] Step 3.2: Design a node representation learning module to learn the node representation of two types of relationship information, including the following steps:

[0160] The process is as follows: The node representation learning module uses the GCN network to learn the node representations of two types of relationship information respectively. and They represent the node representation of co-occurrence relationship and the node representation of succession relationship respectively. Figure 5 As shown, the GCN network uses two GCN layers, each of which represents the nodes learned by the previous layer. or As input, output new node representation after learning or Input node representation of the first GCN layer is the word embedding of the label, where N represents the number of labels, that is, the number of types of sound events in the dataset, and f represents the dimension of the label word embedding; for the last layer (i.e. the second layer), the output is Where o is equal to the dimension of the acoustic features output by the acoustic feature learning module;

[0161] Step 3.2.1: Obtain the sound event co-occurrence relationship graph and the sound event succession relationship graph constructed in step 1 as G = (V, E), where V contains all N nodes in the network, i.e., the N sound events to be detected, v i ∈V, E represents the edge between nodes (v i ,v i )∈E, the eigenvector matrix of the node is Where N is the number of nodes and f is the dimension of the feature vector.

[0162] Step 3.2.2: In order to retain the node's own information while aggregating neighbor information, add self-connections to each node:

[0163]

[0164] Where A represents the adjacency matrix, Element A in ij is a real weight; I N Indicates that each node has a self-connection added. It means that each node has the adjacency matrix after adding the self-connection relationship. Represents the degree matrix after each self-connection is added, the degree matrix D = ∑ j A ij .

[0165] Step 3.2.3: Due to the problem of over-smoothing in the training of multi-layer graph neural networks, the distinguishability of nodes will be reduced. In order to obtain better node representation, reduce the impact of over-smoothing, and maintain sufficient receptive field, this application uses parameter β to adjust the weight ratio of the node itself and its neighbors:

[0166]

[0167] in, Represents the adjacency matrix after adjusting the weight ratio between itself and its neighbors. β is used to allocate the weight ratio between neighbors and itself, β∈[0,1]; the larger the value of β, the greater the weight of neighbors, and the smaller the value of β, the greater the weight of itself.

[0168] Step 3.2.4: The propagation between layers of the ordinary GCN network follows the following formula:

[0169]

[0170] The propagation between layers of the improved GCN network follows the following formula:

[0171]

[0172] Among them, H (l) Represents the input of the l-th layer GCN network, W (l) It represents the parameters of the l-th layer GCN network to be trained, and h() represents the corresponding activation function.

[0173] Step 3.2.5: Construct the first GCN layer and transform the node feature vector matrix Input the GCN network improved by step 3.2.4 above to learn the node representation As shown in the following formula:

[0174]

[0175] Step 3.2.6: Construct the second GCN layer and aggregate the updated node feature vector matrix after the first GCN layer Input it and learn the node representation As shown in the following formula:

[0176] The node representation learned in the last layer As the output of the node representation learning module, the feature sequence of the node representation outputted is N×512, where and They represent the node representation of co-occurrence relationship and the node representation of succession relationship respectively.

[0177] Step 3.3: Transpose the feature sequences N×512 represented by the nodes output by the two types of relationship information learned by GCN to 512×N, and then fuse them in the following two ways to obtain the combined feature sequence:

[0178] Method 1: The two obtained nodes represent the feature sequence of learning output and Perform a corresponding element multiplication, and then perform a corresponding element multiplication with the feature sequence output by acoustic feature learning, combine the two node representation outputs and the acoustic feature output, and obtain the combined feature sequence 512×N;

[0179] Method 2: Two types of nodes represent the output feature sequence and Perform weighted combination and use parameter γ to adjust the weight ratio of the feature sequences of the two nodes to represent the output. The combination is as follows:

[0180]

[0181] in, It represents the feature sequence of nodes with two kinds of relationship information after weighted combination. The parameter γ is used to adjust the weight ratio of the two. Then the node representation after weighted combination is Multiply the corresponding elements with the feature sequence output by acoustic feature learning, combine the two node representation outputs with the acoustic feature output, and obtain the combined feature sequence 512×N;

[0182] Step 3.4: Design a sound event classification network to classify sound events. The sound event classification includes the following steps:

[0183] The process is as follows: the combined feature sequence is sent to the sound event classification network, such as Figure 4 As shown, the sound event classification network consists of two FC layers; the first FC layer contains R nodes, R is 128, and each node has linear activation; the second FC layer consists of N nodes with sigmoid activation, each node corresponds to one of the N sound event categories to be detected, and multiple classes can be activated simultaneously using sigmoid activation, and scores are output. The cross entropy loss function is used for training, and the final sound classification and detection results are output. The output of sound event detection (SED) is in the continuous range of [0,1]. When the output exceeds the threshold of 0.5, it is judged that the sound event exists. For the sound event that is judged to exist, the start and end time and duration of the event are output;

[0184] Step 3.4.1: Create the first FC layer and feed the feature sequence 512×N combined in step 3.3 into it. The first FC layer contains R nodes, R is 128, each node has linear activation, and the output feature sequence is 512×128.

[0185] Step 3.4.2: Create a second FC layer and input the feature sequence 512×128 output by the first FC layer. The second FC layer consists of N nodes with sigmoid activation, each node corresponds to one of the N sound event categories to be detected, and the output is 512×N. Using sigmoid activation, multiple classes can be activated at the same time and the scores can be output. Use the cross entropy loss function for training and output the final sound classification result. The SED output is in the continuous range of [0,1]. When the output exceeds the threshold of 0.5, the sound event is judged to exist;

[0186] Step 3.4.3: Output the sound event category, start and end time, and duration determined in step 3.4.2;

[0187] The model structure of the sound event classification network is as follows Figure 5 shown.

[0188] Step 4: Design a volume-based rapid warning module for sound event warning, including the following steps:

[0189] Step 4.1: Determine the amplitude of the sound wave and obtain the volume at the current moment using the amplitude value of the audio signal obtained in step 2.3.2. The amplitude value is recorded as V t , the value of t is the current moment, ranging from 0 to the last moment of the audio signal, and the interval is 100 milliseconds;

[0190] Step 4.2: Determine the volume threshold based on the sound event detection model and statistical analysis;

[0191] The steps to obtain the volume threshold are as follows:

[0192] 1) Use audio data to train the sound event detection model;

[0193] 2) Use the trained sound event detection model to perform sound event detection on a large amount of audio data;

[0194] 3) According to the prediction results of the sound event detection model, the volume amplitude values ​​of different sound event categories are counted and risk assessment and classification are performed according to the sound event categories. Subsequently, a large amount of data is statistically analyzed to select the optimal classification threshold Y.

[0195] Step 4.3: Based on the acquired amplitude value and volume threshold, realize the warning based on the high volume;

[0196] During normal driving, the volume is usually within a certain range. When a loud noise suddenly occurs, it must mean that an emergency event with a loud volume has occurred within a certain distance of the vehicle, and this emergency event is likely to indicate a crisis.

[0197] The real-time amplitude value V obtained in step 4.1 and step 4.2 t And the corresponding volume threshold Y can be used to simply and efficiently carry out crisis warning. The volume threshold Y is used for two levels. When V t When ≤Y, it is considered as risk-free situation, V t >Y, a quick risk warning is issued, such as Figure 6 as shown in .

[0198] This early warning technology is used in smart cars and connected to the on-board smart systems, such as Figure 7 When a loud noise is suddenly received, the warning technology will make a judgment in a very short time and transmit the crisis warning information to the vehicle-mounted intelligent central control system. The system will urgently make the most common crisis response measures such as deceleration to ensure that there is enough time to confirm the crisis situation and respond. Then the vehicle-mounted intelligent audio monitoring system will analyze and detect the sound event in a relatively short time to obtain the specific crisis situation, and make more reasonable response measures according to the specific crisis situation.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for determining a driving event based on sound signals, characterized in that: The following steps are involved: Step 1: Construct a sound event co-occurrence relationship graph and a sound event succession relationship graph; Step 2: Preprocess the audio data in the sound event dataset and extract feature sequences; Step 3: Design a sound event detection model and perform sound event detection, including the following steps: Step 3.1: Design an acoustic feature learning module to perform acoustic feature learning; the acoustic feature learning module includes a convolutional neural network, a bidirectional RNN layer and two FC layers; the convolutional neural network uses a three-layer two-dimensional CNN to learn the acoustic features in the feature sequence and inputs them into the bidirectional RNN layer; the bidirectional RNN layer is used to learn the temporal context information from the CNN output and input it into the two FC layers; the two FC layers further extract the acoustic features of the sound event; Step 3.2: Design a node representation learning module to learn the node representation of two types of relationship information, including the following steps: The node representation learning module uses the GCN network to learn the node representations of co-occurrence and successive relationship information respectively, where and They represent the node representation of co-occurrence relationship and the node representation of succession relationship respectively; the GCN network uses two GCN layers, each GCN layer represents the node learned by the previous layer or As input, output new node representation after learning or Input node representation of the first GCN layer is the word embedding of the label, where N represents the number of labels, that is, the number of types of sound events in the dataset, and f represents the dimension of the label word embedding; for the last layer, the output is Where o is equal to the dimension of the acoustic features output by the acoustic feature learning module; Step 3.3: Transpose the feature sequences of the nodes represented by the two types of relationship information learned by GCN, and then fuse them in the following two ways to obtain the combined feature sequence: Method 1: The two obtained nodes represent the feature sequence of learning output and Perform a corresponding element multiplication, and then perform a corresponding element multiplication with the feature sequence output by acoustic feature learning, combine the two node representation outputs and the acoustic feature output, and obtain the combined feature sequence; Method 2: Two types of nodes represent the output feature sequence and Perform weighted combination and use parameter γ to adjust the weight ratio of the feature sequences of the two nodes to represent the output. The combination is as follows: in, It represents the feature sequence of nodes with two kinds of relationship information after weighted combination. The parameter γ is used to adjust the weight ratio of the two. Then the node representation after weighted combination is Multiply the corresponding elements with the feature sequence output by acoustic feature learning, combine the two node representation outputs with the acoustic feature output, and obtain the combined feature sequence; Step 3.4: Design a sound event classification network to classify sound events, including the following steps: The combined feature sequence is sent to a sound event classification network for sound event classification; the sound event classification network is composed of two FC layers; the first FC layer contains R nodes, each of which has linear activation; the second FC layer is composed of N nodes with sigmoid activation, each of which corresponds to one of the N sound event categories to be detected, and multiple classes can be activated simultaneously using sigmoid activation, and scores are output, and the cross entropy loss function is used for training, and the final sound classification and detection results are output, and the output of the sound event detection is in the continuous range of [0,1]. When the output exceeds the threshold, it is judged that the sound event exists, and the start and end time and duration of the sound event that is judged to exist are output; Step 4: Design a volume-based rapid warning module for sound event warning, including the following steps: Step 4.1: Determine the amplitude of the sound wave and obtain the volume at the current moment using the amplitude value of the audio signal obtained in step 2.3.

2. The amplitude value is recorded as V t , the value of t is the current moment; Step 4.2: Determine the volume threshold based on the sound event detection model and statistical analysis; Step 4.3: Based on the acquired amplitude value and volume threshold, realize the warning based on the high volume; Use the volume threshold Y to perform two classifications. When the amplitude value V of the audio signal t When ≤Y, it is considered as risk-free situation, V t A quick risk warning will be issued when ≥Y.

2. The method for determining a driving event based on sound signals according to claim 1, characterized in that: The step 1 comprises: Step 1.1: Obtain and process the label information of the sound event; Step 1.2: Design the sound event association window; A sound event association window of length T is used at the end time of each sound event in the audio in the sound event dataset to obtain other sound events that occur successively after each sound event, thereby achieving the detection performance of the successive occurrence of sound events; Step 1.3: Calculate the sound event relationship and construct the sound event co-occurrence relationship graph and the succession relationship graph, including the following steps: Step 1.3.1: Obtain the sound event information of the co-occurrence relationship; Step 1.3.2: Obtaining sound event information of successive association relationships; Step 1.3.3: Obtain the number of occurrences of each type of sound event in the sound event dataset; Count the number of occurrences of various sound events in the sound event dataset, denoted as X i , indicating the sound event L i Number of occurrences; Step 1.3.4: Calculate the probability of simultaneous and successive occurrence of various sound events in the sound event data set through the prior probability formula; take each type of sound event as a node, and use the calculated probabilities as the directed edges of the sound event co-occurrence relationship graph and the sound event succession relationship graph to connect these nodes, and complete the construction of the sound event co-occurrence relationship graph and the sound event succession relationship graph.

3. The method for determining a driving event based on sound signals according to claim 2, characterized in that: The step 1.1 comprises: Read the label file in the sound event dataset to obtain the label information of the training dataset, including the sound event category in the label file of each audio, and the start and end time of each sound event; Each type of sound event is denoted as L i , i = 0, 1, 2, 3...n, n is the total number of sound event categories, and the start and end time of each sound event is recorded, where each sound event is recorded as M is the total number of sound events, and the start time of each sound event is expressed as The end time is expressed as 4. The method for determining a driving event based on sound signals according to claim 3, characterized in that: Step 1.3.1: each sound event obtained in step 1.1 Start and end time and Get every sound event The co-occurrence range Other sound events within a sound event When there are other sound events within the co-occurrence range of j=0,1,2,3...n, and i≠j; the number of times the sound event co-occurs is recorded as Indicates sound event L i With sound events L j The number of times they occur at the same time, that is, the number of co-occurrences.

5. The method for determining a driving event based on sound signals according to claim 4, characterized in that: Step 1.3.2: Each sound event obtained in step 1.1 Start and end time and Combined with the length T of the sound event association window, obtain each sound event The sound event correlation window range Other sound events within a sound event If there are other sound events within the sound event association window, it is considered as a sequential relationship, recorded as The number of times two consecutive sound events occur in succession is recorded as Indicates sound event L i With sound events L j The number of times a sound event occurs within the same sound event window.

6. The method for determining a driving event based on sound signals according to claim 5, characterized in that: The step 1.3.4 is based on the number of occurrences of each type of sound event in the obtained sound event data set X i and the number of times the audio events co-occur Calculate when the sound event L i When the sound event L occurs j Probability As shown in the following formula: in, For the sound event L i When the sound event L occurs j The probability of is the number of co-occurrences of audio events, X i is the number of occurrences of various sound events in the sound event dataset; According to the number of occurrences of various events X i and the number of times the events occurred in succession Calculate the probability of sound events happening one after another As shown in the following formula: The probability thresholds α1, α2, α1, α2∈(0,1) are used to filter the sound events, and the sound events with probabilities lower than the probability thresholds are used as the relationship weights between nodes, and the relationship weights are reset to 0, as shown in the following formula:

7. The method for determining a driving event based on sound signals according to claim 1, characterized in that: The step 2 uses a 512-point fast Fourier transform to extract a spectrogram from each C channel of the multi-channel audio on a Hanning window with a length of 512 and an overlap of 50%; then extracts the phase and amplitude of the spectrogram, and uses the extracted phase and amplitude as separate features, wherein the amplitude value is also used to determine the volume in the early warning technology, and includes the following steps: Step 2.1: Set the audio sampling frequency and process the audio data into frames; Step 2.2: Select a window function to perform a frame-by-frame windowing operation on the framed audio data obtained in step 1; Step 2.3: Obtain the spectrum and extract the phase and amplitude; Step 2.3.1: Convert the framed and windowed audio data from the time domain to the frequency domain through fast Fourier transform to obtain the frequency domain signal of the audio data Step 2.3.2: Perform a modulo operation on the frequency domain signal obtained in step 2.3.1 to obtain the amplitude spectrum of the audio data; Step 2.3.3: Perform a phase angle operation on the frequency domain signal obtained in step 2.3.1 to obtain the phase spectrum of the audio data Step 2.3.4: Combine the amplitude spectrum and phase spectrum obtained in step 2.3.2 and step 2.3.3 to obtain the characteristic sequence.

8. The method for determining a vehicle driving event based on sound signals according to claim 7, characterized in that: In step 3.1, the feature sequence obtained by preprocessing in step 2 is sent as input to the convolutional neural network; in the convolutional neural network part, a three-layer two-dimensional CNN is used to learn the acoustic features of the feature sequence; The first CNN layer has B1 filters with a 3×3×2C-dimensional receptive field, where B1 is 256 and C is 4. The number of filters in the next two CNN layers is 128 and 64 respectively. The step size and padding of the three CNN layers are both 1 to ensure that the feature sequence length T remains unchanged. After each CNN layer, the output is normalized using batch normalization and activated using the ReLU function, and then the maximum pooling along the frequency axis is used to reduce the dimension, thereby keeping the feature sequence length T unchanged. The three CNN layers use filter kernels that span all channels, allowing the CNN to learn relevant intra-channel features in the time and frequency dimensions; the filter output of the last CNN layer is T×2×B3, i.e. 512×2×64, where the frequency dimension is reduced to 2 due to the effect of the maximum pooling layer; after each of the three CNN layers, the maximum pooling in frequency is 8, 4, 4 respectively; the feature sequence 512×2×64 output by the three CNN layers is swapped in dimension and further reshaped into a T-frame sequence of feature vectors of length 2B3, i.e. 512×128, and fed into a bidirectional RNN layer, which is used to learn temporal context information from the CNN output; A gated recurrent unit with Q nodes is used in each RNN layer. The gated recurrent unit is used to extract the temporal context information of the sound event to obtain the timing information of various sound events in the audio. The value of Q is 128, and the tanh function is used for activation. The dimension of the feature sequence output is consistent with the input. The feature sequence output by the bidirectional RNN layer is then sent to two FC layers to further extract the acoustic features of the sound event. The first FC layer contains R nodes, R is 128, and each node has linear activation; the second FC layer consists of N nodes, and the value of N is the number of sound event categories in the corresponding data set; the feature sequence of the final output acoustic features is 512×N; including the following steps: Step 3.1.1: Create the first two-dimensional CNN layer. The input of this layer is the feature sequence 512×256×8 output by step 2 preprocessing. Use the filter kernel across all channels to allow CNN to learn relevant intra-channel features in the time and frequency dimensions. This layer has B1 filters with 3×3×2C-dimensional receptive fields, B1 is 256, C is 4, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; after passing through the CNN layer, the feature sequence becomes 512×256×256; use batch normalization to standardize the CNN layer output and activate it with the ReLU function, then use the maximum pooling along the frequency axis to reduce the dimension to 512×32×256, thereby keeping the feature sequence length T unchanged, and the maximum pooling on the frequency is 1×8; Step 3.1.2: Create a second two-dimensional CNN layer, which has B2 filters with 3×3×8 dimensional receptive fields, B2 is 128, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; the input of this layer is the feature sequence 512×32×256 output by the first CNN layer, which first passes through the convolution layer. After convolution, the feature sequence dimension becomes 512×32×128, and then the output is standardized using batch normalization and activated using the ReLU function. After that, the maximum pooling along the frequency axis is reduced to 512×8×128, and the maximum pooling on the frequency is 1×4; Step 3.1.3: Create the third two-dimensional CNN layer, which has B3 filters with 3×3×8 dimensional receptive fields, B3 is 64, and the step size and padding are both 1 to ensure that the feature sequence length T remains unchanged; the input of this layer is the feature sequence 512×8×128 output by the second CNN layer. First, it passes through the convolution layer. After the convolution, the feature sequence dimension becomes 512×8×64, and then the output is standardized using batch normalization and activated using the ReLU function. After that, the maximum pooling along the frequency axis is used to reduce the dimension to 512×2×64, and the maximum pooling on the frequency is 1×4; Step 3.1.4: The feature sequence 512×2×64 output by the three-layer CNN layer is swapped into 2×512×64 and further reshaped into a T-frame sequence of feature vectors of length 2B3, i.e. 512×128; Step 3.1.5: Create two bidirectional RNN layers and feed the feature sequence reshaped in step 3.1.4 into them to extract the temporal context information of the sound event. Each RNN layer uses a gated recurrent unit with Q nodes, Q is 128, and is activated by the tanh function. The dimension of the final output feature sequence is consistent with the input, that is, 512×128; Step 3.1.6: Create two FC layers and send the feature sequence of the sound event temporal context information extracted by the RNN layer into them for further training; the first FC layer contains R nodes, R is 128, and each node has linear activation; the second FC layer consists of N nodes, and the value of N is the number of sound event categories in the corresponding data set; the final output feature sequence of acoustic features is 512×N.

9. The method for determining a vehicle driving event based on sound signals according to claim 8, characterized in that: The step 3.2 comprises: Step 3.2.1: Obtain the sound event co-occurrence relationship graph and the sound event succession relationship graph constructed in step 1 as G = (V, E), where V contains all N nodes in the network, that is, the N sound events to be detected, v i ∈V, E represents the edge between nodes (v i ,v i )∈E, the eigenvector matrix of the node is Where N is the number of nodes and f is the dimension of the feature vector; Step 3.2.2: In order to retain the node's own information while aggregating neighbor information, add self-connections to each node: Where A represents the adjacency matrix, Element A in ij is a real weight; I N Indicates that each node has a self-connection added. It means that each node has the adjacency matrix after adding the self-connection relationship. Represents the degree matrix after each self-connection is added, the degree matrix D = ∑ j A ij ; Step 3.2.3: Use parameter β to adjust the weight ratio between the node itself and its neighbors: in, Represents the adjacency matrix after adjusting the weight ratio between itself and its neighbors. β is used to allocate the weight ratio between neighbors and itself, β∈[0,1]; Step 3.2.4: The propagation of the improved GCN network between layers follows the following formula: Among them, H (l) represents the input of the l-th layer GCN network, W (l) represents the parameters to be trained of the l-th layer GCN network, and h() represents the corresponding activation function; Step 3.2.5: Construct the first GCN layer and transform the node feature vector matrix Input the GCN network improved by step 3.2.4 above to learn the node representation As shown in the following formula: Step 3.2.6: Construct the second GCN layer and aggregate the updated node feature vector matrix after the first GCN layer Input it and learn the node representation As shown in the following formula: The node representation learned in the last layer As the output of the node representation learning module, the feature sequence of the node representation outputted is N×512, where and They represent the node representation of co-occurrence relationship and the node representation of succession relationship respectively.

10. The method for determining a driving event based on sound signals according to claim 9, characterized in that: The specific method of determining the volume threshold based on the sound event detection model and statistical analysis in step 4.2 is: 1) Use audio data to train the sound event detection model; 2) Use the trained sound event detection model to perform sound event detection on the audio data; 3) According to the detection results of the sound event detection model, the volume amplitude values ​​of different sound event categories are counted and risk assessment and classification are performed according to the sound event categories. Subsequently, a large amount of data is statistically analyzed to select the optimal classification threshold Y.