Vehicle identification system and method based on audio data and vibration data

Abstract

The invention discloses a vehicle recognition system and method based on audio data and vibration data. The method comprises the following steps: training an initial vehicle recognition model by using vehicle audio data, and performing incremental updating on the trained vehicle recognition model by using the vehicle audio data which cannot be recognized; collecting vehicle vibration data and judging a vibration energy threshold value; collecting single-frame vehicle audio data and judging an audio energy threshold value; and performing pattern recognition on the collected vehicle audio data by using the vehicle recognition model after incremental updating, outputting a recognition result, and uploading the vehicle audio data which cannot be recognized to a server. According to the method, the model parameters are incrementally updated by introducing new acquired data, so that the adaptability of the vehicle identification method to the vehicle audio data change is ensured, and the accuracy and advancement of vehicle identification are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent transportation perception, and particularly relates to a vehicle recognition system and method based on audio data and vibration data. Background Art

[0002] With the development of smart cities and intelligent transportation systems, vehicle recognition technology has become a core requirement for road environment perception. Traditional vehicle recognition means mainly rely on technologies such as video surveillance, geomagnetic sensors, radars, and vibration sensors. Although these technologies can achieve vehicle monitoring and recognition to a certain extent, there are some limitations in practical applications; for example, the vehicle recognition technology based on video surveillance can obtain rich information, but there are problems such as privacy disputes, decreased detection performance at night, high computing power requirements, and failure in bad weather; although the detection accuracy of geomagnetic sensors and radar systems is relatively high, they are greatly affected by environmental interference, and at the same time, the installation and maintenance costs are relatively high; vibration sensors can well sense whether a vehicle passes by, but they are sensitive to the deployment location and it is difficult to distinguish vehicle types.

[0003] In recent years, vehicle recognition technology based on audio data has gradually attracted attention. The acoustic features such as engine noise and tire noise generated during vehicle driving are unique and distinguishable, and microphone sensors have the advantages of easy deployment, low cost, and strong environmental adaptability.

[0004] However, the existing vehicle recognition technology based on audio data still has defects. Traditional acoustic detection needs to continuously collect audio signals and perform real-time processing, resulting in high device power consumption and being difficult to meet the long-term deployment requirements of battery-powered scenarios; most solutions can only achieve vehicle presence detection, and lack the ability to finely identify vehicle types; existing embedded devices are limited by computing power and are difficult to run complex machine learning models under low power consumption constraints.

[0005] In view of the above problems, there is an urgent need for an acoustic detection solution that takes into account low power consumption, high robustness, and supports multi-dimensional vehicle feature recognition to promote the low-cost large-scale application of intelligent transportation perception technology. Summary of the Invention

[0006] Aiming at the deficiencies of the above-mentioned existing technologies, the purpose of the present invention is to provide a vehicle recognition system and method based on audio data and vibration data to solve the problem that there is no acoustic detection solution with low power consumption, high robustness, and supporting multi-dimensional vehicle feature recognition in the existing technologies.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A vehicle recognition system based on audio data and vibration data of the present invention includes: a server and an edge device; wherein,

[0009] The server includes: a model training module and an incremental update module;

[0010] The model training module is used to train an initial vehicle recognition model using vehicle audio data to obtain a trained vehicle recognition model;

[0011] The incremental update module is used to incrementally update the trained vehicle recognition model using the vehicle audio data uploaded by the edge device to obtain an incrementally updated vehicle recognition model;

[0012] The edge device includes: a vibration data acquisition module, an audio data acquisition module, and a vehicle recognition module;

[0013] The vibration data acquisition module is used to acquire vehicle vibration data and determine the vibration energy threshold;

[0014] The audio data acquisition module is used to acquire vehicle audio data and determine the audio energy threshold;

[0015] The vehicle recognition module is used to perform pattern recognition on the vehicle audio data using the incrementally updated vehicle recognition model and send the unrecognized vehicle audio data to the incremental update module.

[0016] The present invention also provides a vehicle recognition method based on audio data and vibration data. Based on the above system, the steps are as follows:

[0017] 1) Train an initial vehicle recognition model using vehicle audio data and incrementally update the trained vehicle recognition model using the unrecognized vehicle audio data;

[0018] 2) Acquire vehicle vibration data and determine the vibration energy threshold. If it is less than the vibration energy threshold, repeat step 2), otherwise go to step 3);

[0019] 3) Acquire a single frame of vehicle audio data and determine the audio energy threshold. If it is less than the audio energy threshold, return to step 2), otherwise, continuously acquire multiple frames of vehicle audio data and go to step 4);

[0020] 4) Perform pattern recognition on the vehicle audio data acquired in step 3) using the incrementally updated vehicle recognition model in step 1) and output the recognition result, and upload the unrecognized vehicle audio data to the server.

[0021] Further, step 1) specifically includes:

[0022] 11) Train the initial vehicle recognition model M1 using the vehicle audio dataset D1;

[0023] The vehicle audio dataset D1 includes multiple labeled vehicle audio samples. Each vehicle audio sample is derived from the audio data composed of engine noise and tire noise collected when the vehicle passes through the collection point. The corresponding label is the vehicle type, including cars, buses, and trucks. The expression of the vehicle audio dataset D1 is as follows:

[0024] D1 = {d1, d2, …, d n}

[0025] d i 0 i , y i}

[0026] where n is the total number of dataset samples, d i is the i-th sample, x i is the vehicle audio data corresponding to the i-th sample, and y i is the vehicle type corresponding to the i-th sample;

[0027] The initial vehicle recognition model M1 is a classification model based on a deep convolutional neural network. Its input is vehicle audio data, and its output is the vehicle type with the highest classification probability. The model M1 specifically includes: an input layer, a preprocessing layer, a convolutional layer, a pooling layer, a fully connected layer, and a Softmax output layer. The model M1 is supervised-trained based on the vehicle audio dataset D1. The training process uses the backpropagation algorithm and the gradient descent optimization algorithm to minimize the cross-entropy loss function between the classification output and the true label. The trained vehicle recognition model M2 can realize the mapping from vehicle audio data to vehicle type, and the expression is as follows:

[0028]

[0029] where, is the input space, is the output space, x is a specific vehicle audio data in the input space, and θ is the set of trainable parameters of the model;

[0030] 12) Perform incremental update on the trained vehicle recognition model M2 to obtain the incrementally updated vehicle recognition model M3. Specifically: Receive the difficult sample set D hard = {S (1) , S (2) , …, S (K)} sent by the edge device, where K is the total number of samples in the difficult sample set, and S (i) is the i-th sample, and each sample is vehicle audio data; Annotate S (i) to obtain the corresponding vehicle type y i , and construct a new sample as Then perform federated aggregation to update the trained vehicle recognition model M2 as follows:

[0031]

[0032] where θ new is the set of trainable parameters of the vehicle recognition model M3 after incremental update, θ old is the set of trainable parameters of the trained vehicle recognition model M2, η is the learning rate, is the gradient, and L is the cross-entropy loss function;

[0033] The edge device obtains the parameter set θ new , and adopts a smoothing update strategy to prevent the vehicle recognition model M3 after incremental update from oscillating, ensuring that the model retains its original feature expression ability while adapting to new data. The expression of the smoothing update strategy is as follows:

[0034] θ new = λθ new + (1 - λ)θ old

[0035] where λ is the weight coefficient, and λ ∈ [0, 1].

[0036] Furthermore, step 2) specifically includes:

[0037] Collect vehicle vibration data to obtain vibration data of E1 × T1, where E1 and T1 respectively represent the vibration energy and duration of the vibration data. The calculation method of E1 is as follows:

[0038]

[0039] where a[n] is the discrete vibration data collected by the vibration sensor, and L1 is the number of vibration sampling points within the duration T1; judge whether the average vibration energy is less than the preset average vibration energy threshold E′1, or whether the vibration duration T1 is less than the vibration duration threshold T′1, If so, repeat step 2); otherwise, enter step 3).

[0040] Furthermore, step 3) specifically includes:

[0041] Collect single-frame vehicle audio data to obtain audio data S1 of E2 × T2, where E2 and T2 respectively represent the audio energy and frame length of the single-frame audio data. The calculation method of E2 is as follows:

[0042]

[0043] Among them, x[n] is the discrete audio data collected by the microphone, and L2 is the number of sampling points of each frame of audio data; determine the average audio energy whether it is less than the preset average audio energy threshold E′2, if so, return to step 2); otherwise, continuously collect a total of N frames of vehicle audio data to obtain E N ×T N of the audio data S N , where E N represents the total energy of S N , T N represents the total duration of S N , T N = N*T2, enter step 4).

[0044] Furthermore, step 4) specifically includes:

[0045] Short-time pattern recognition: Use the incrementally updated vehicle recognition model M3 to perform pattern recognition on the single-frame vehicle audio data collected in step 3). If the short-time classification probability is greater than or equal to the short-time classification probability threshold, output the recognition result; otherwise, perform long-time pattern recognition;

[0046] Long-time pattern recognition: Use the updated vehicle recognition model M3 to perform pattern recognition on the multi-frame vehicle audio data collected in step 3). If the long-time classification probability is greater than or equal to the long-time classification probability threshold, output the recognition result; otherwise, increase the number of audio data frames and repeat the long-time pattern recognition; when the number of repetitions reaches the upper limit, save the audio data to the difficult sample set; when the number of samples in the difficult sample set reaches the upper limit, upload the difficult sample set to the server.

[0047] Furthermore, the short-time pattern recognition specifically includes:

[0048] Use the incrementally updated vehicle recognition model M3 to perform short-time pattern recognition on the audio data S1 to obtain the vehicle recognition result R1 and its corresponding short-time classification probability P1; the vehicle recognition result R1 is the output of the incrementally updated vehicle recognition model M3, that is, the vehicle type with the highest model classification probability; the short-time classification probability P1 is the confidence of the incrementally updated vehicle recognition model M3 in the recognition result R1, and the calculation formula is as follows:

[0049]

[0050] where E mel_1 is the Mel cepstrum coefficient energy, X[k] is the frequency domain energy of the audio data S1, H m [k] is the parameter of the mth Mel filter, M is the total number of Mel filters, zi is the probability of the i-th category of the model output, C is the total number of categories for classification, and e is the base of the natural logarithm. At the same time, the dynamic adjustment of the short-time classification probability threshold P′1 is as follows:

[0051] P′1 = μ hist + k·σ hist

[0052] where k is the standard deviation adjustment hyperparameter, k ∈ (0, 1), μ hist and σ hist are respectively the moving window mean and standard deviation based on historical classification probabilities, as follows:

[0053]

[0054]

[0055] where W is the window length, is the recognition probability at the t-th historical time. If the short-time classification probability P1 is greater than or equal to the short-time classification probability threshold P′1, then a vehicle recognition result R1 is output; otherwise, long-time pattern recognition is performed.

[0056] Furthermore, the long-time pattern recognition specifically includes:

[0057] 41) Use the incrementally updated vehicle recognition model M3 to perform long-time pattern recognition on the audio data S l to obtain the vehicle recognition result R l and its corresponding long-time classification probability P l ; the audio data S l is the first l frames of audio data collected in step 3), where l ∈ {2, 3,..., N}; the vehicle recognition result R l is the output of the incrementally updated vehicle recognition model M3, that is, the vehicle type with the highest model classification probability; the short-time classification probability P l is the confidence of the incrementally updated vehicle recognition model M3 in the vehicle recognition result R l , and the calculation method is as follows:

[0058]

[0059] where E mel_l is the Mel cepstral coefficient energy, X l [k] is the frequency domain energy of the audio data S l , H m [k] is the parameter of the m-th Mel filter, z i is the probability of the i-th category of the model output, C is the total number of categories for classification, and e is the base of the natural logarithm. At the same time, the long-time classification probability threshold P′l The dynamic adjustment is as follows:

[0060] P′ l = μ hist + k·σ hist

[0061] Judge whether the long - term classification probability P l is greater than or equal to the long - term classification probability threshold P′ l , if so, output a vehicle recognition result R l ; otherwise, let l = l + 1, judge whether l is less than or equal to N, if so, repeat step 41); otherwise, save the audio data S N to the difficult sample set D hard and execute step 42);

[0062] 42) Output that the vehicle recognition result is unrecognizable, and judge whether the number of samples in the difficult sample set D hard is equal to the set upper limit K of the number of samples. If so, upload the difficult sample set to the server, clear the locally saved difficult sample set and return to step 2); otherwise, directly return to step 2).

[0063] Advantages of the present invention:

[0064] 1. By introducing new collected data to incrementally update the model parameters, the present invention ensures the adaptability of the vehicle recognition method to the changes in vehicle audio data, thus realizing the accuracy and advancement of vehicle recognition;

[0065] 2. The present invention uses vehicle vibration data and vehicle audio data for threshold detection, and performs vehicle recognition only when the vehicle passes by, realizing low - power operation;

[0066] 3. The present invention combines short - term pattern recognition and long - term pattern recognition for vehicle recognition, can capture both short - term burst features and long - term stable features in vehicle audio data, and thus accurately gives the recognition result; in addition, by automatically collecting difficult samples, the present invention further optimizes the recognition accuracy and robustness of the vehicle recognition model. Description of the Drawings

[0067] Figure 1 is a schematic structural diagram of the system of the present invention;

[0068] Figure 2 is a flowchart of the method of the present invention. Detailed Embodiments [[ID=�3]]

[0069] For the convenience of those skilled in the art to understand, the present invention is further described below in conjunction with embodiments and drawings. The content mentioned in the embodiments does not limit the present invention.

[0070] Refer to Figure 1 As shown, a vehicle recognition system based on audio data and vibration data of the present invention includes: a server and an edge device; wherein,

[0071] The server includes: a model training module and an incremental update module;

[0072] The model training module is used to train an initial vehicle recognition model using vehicle audio data to obtain a trained vehicle recognition model;

[0073] The incremental update module is used to incrementally update the trained vehicle recognition model using the vehicle audio data uploaded by the edge device to obtain an incrementally updated vehicle recognition model;

[0074] The edge device includes: a vibration data acquisition module, an audio data acquisition module, and a vehicle recognition module;

[0075] The vibration data acquisition module is used to acquire vehicle vibration data and determine the vibration energy threshold;

[0076] The audio data acquisition module is used to acquire vehicle audio data and determine the audio energy threshold;

[0077] The vehicle recognition module is used to perform pattern recognition on the vehicle audio data using the incrementally updated vehicle recognition model and send the vehicle audio data that cannot be recognized to the incremental update module.

[0078] Refer to Figure 2 As shown, the present invention also provides a vehicle recognition method based on audio data and vibration data. Based on the above system, the steps are as follows:

[0079] 1) Train an initial vehicle recognition model using vehicle audio data and incrementally update the trained vehicle recognition model using the vehicle audio data that cannot be recognized; specifically including:

[0080] 11) Train the initial vehicle recognition model M1 using the vehicle audio data set D1;

[0081] The vehicle audio data set D1 includes multiple labeled vehicle audio samples. Each vehicle audio sample is derived from the audio data composed of engine noise and tire noise collected when the vehicle passes through the collection point, and the corresponding label is the vehicle type, including cars, buses, and trucks; the expression of the vehicle audio data set D1 is as follows:

[0082] D1 = {d1, d2, …, d n}

[0083] d i = {x i , y i}

[0084] Among them, n is the total number of samples in the dataset, d i is the i-th sample, x i is the vehicle audio data corresponding to the i-th sample, y i is the vehicle type corresponding to the i-th sample;

[0085] The initial vehicle recognition model M1 is a classification model based on a deep convolutional neural network, with its input being vehicle audio data and its output being the vehicle type with the highest classification probability; the model M1 specifically includes: an input layer, a preprocessing layer, a convolutional layer, a pooling layer, a fully connected layer, and a Softmax output layer; the model M1 is supervised-trained based on the vehicle audio dataset D1, and the backpropagation algorithm and the gradient descent optimization algorithm are used during the training process to minimize the cross-entropy loss function between the classification output and the true label; the trained vehicle recognition model M2 can realize the mapping from vehicle audio data to vehicle types, and the expression is as follows:

[0086]

[0087] Among them, is the input space, is the output space, x is a specific vehicle audio data in the input space, and θ is the set of trainable parameters of the model;

[0088] 12) Perform incremental update on the trained vehicle recognition model M2 to obtain the incrementally updated vehicle recognition model M3. Specifically: receive the difficult sample set D hard ={S (1) , S (2) , …, S (K)} sent by the edge device, where K is the total number of samples in the difficult sample set, and S (i) is the i-th sample, and each sample is vehicle audio data; annotate S (i) to obtain the corresponding vehicle type y i , and construct a new sample as Then perform federated aggregation to update the trained vehicle recognition model M2 as follows:

[0089]

[0090] Among them, θ new is the set of trainable parameters of the incrementally updated vehicle recognition model M3, θ old is the set of trainable parameters of the trained vehicle recognition model M2, η is the learning rate, is the gradient, and L is the cross-entropy loss function;

[0091] The edge device obtains the parameter set θnew Adopt a smooth update strategy to prevent the vehicle recognition model M3 from oscillating after incremental update, ensuring that the model retains its original feature expression ability while adapting to new data. The expression of the smooth update strategy is as follows:

[0092] θ new = λθ new +(1 - λ)θ old

[0093] where λ is the weight coefficient and λ ∈ [0, 1].

[0094] 2) Collect vehicle vibration data and determine the vibration energy threshold. If it is less than the vibration energy threshold, repeat step 2); otherwise, proceed to step 3).

[0095] Specifically, step 2) includes:

[0096] Collect vehicle vibration data to obtain vibration data of E1 × T1, where E1 and T1 represent the vibration energy and duration of the vibration data respectively. The calculation method of E1 is as follows:

[0097]

[0098] where a[n] is the discrete vibration data collected by the vibration sensor and L1 is the number of vibration sampling points within the duration T1; judge whether the average vibration energy is less than the preset average vibration energy threshold E′1, or whether the vibration duration T1 is less than the vibration duration threshold T1′. If so, repeat step 2); otherwise, proceed to step 3).

[0099] 3) Collect single-frame vehicle audio data and determine the audio energy threshold. If it is less than the audio energy threshold, return to step 2); otherwise, continuously collect multiple frames of vehicle audio data and proceed to step 4).

[0100] Specifically, step 3) includes:

[0101] Collect single-frame vehicle audio data to obtain audio data S1 of E2 × T2, where E2 and T2 represent the audio energy and frame length of the single-frame audio data respectively. The calculation method of E2 is as follows:

[0102]

[0103] where x[n] is the discrete audio data collected by the microphone and L2 is the number of sampling points for each frame of audio data; judge whether the average audio energy is less than the preset average audio energy threshold E′2. If so, return to step 2); otherwise, continuously collect vehicle audio data for a total of N frames to obtain the audio data S of N ×T N , where E N represents the total energy of S N , and N T represents the total duration of S N , and T N = N * T2, then enter step 4). N

[0104] 4) Use the vehicle recognition model updated incrementally in step 1) to perform pattern recognition on the vehicle audio data collected in step 3) and output the recognition result, and upload the vehicle audio data that cannot be recognized to the server; specifically including:

[0105] Short-time pattern recognition: Use the incrementally updated vehicle recognition model M3 to perform pattern recognition on a single frame of vehicle audio data collected in step 3). If the short-time classification probability is greater than or equal to the short-time classification probability threshold, output the recognition result; otherwise, perform long-time pattern recognition;

[0106] Long-time pattern recognition: Use the updated vehicle recognition model M3 to perform pattern recognition on multiple frames of vehicle audio data collected in step 3). If the long-time classification probability is greater than or equal to the long-time classification probability threshold, output the recognition result; otherwise, increase the number of audio data frames and repeat the long-time pattern recognition; when the number of repetitions reaches the upper limit, save the audio data to the difficult sample set; when the number of samples in the difficult sample set reaches the upper limit, upload the difficult sample set to the server.

[0107] Specifically, the short-time pattern recognition specifically includes:

[0108] Use the incrementally updated vehicle recognition model M3 to perform short-time pattern recognition on the audio data S1 to obtain the vehicle recognition result R1 and its corresponding short-time classification probability P1; the vehicle recognition result R1 is the output of the incrementally updated vehicle recognition model M3, that is, the vehicle type with the highest model classification probability; the short-time classification probability P1 is the confidence of the incrementally updated vehicle recognition model M3 in the recognition result R1, and the calculation formula is as follows:

[0109]

[0110]

[0111] where E mel_1 is the Mel cepstral coefficient energy, X[k] is the frequency domain energy of the audio data S1, and H m ​[k] is the parameter of the m-th Mel filter, M is the total number of Mel filters, and z i is the probability of the i-th category output by the model, C is the total number of classification categories, and e is the base of the natural logarithm; at the same time, the dynamic adjustment of the short-time classification probability threshold P′1 is as follows:

[0112] P′1 = μ hist + k·σ hist

[0113] where k is the standard deviation adjustment hyperparameter, k ∈ (0, 1), μ hist and σ hist are the moving window mean and standard deviation based on historical classification probabilities respectively, as follows:

[0114]

[0115] where W is the window length, is the recognition probability at the t-th historical time; if the short-time classification probability P1 is greater than or equal to the short-time classification probability threshold P′1, then a vehicle recognition result R1 is output, otherwise the long-time pattern recognition is executed.

[0116] Specifically, the long-time pattern recognition specifically includes:

[0117] 41) Use the incrementally updated vehicle recognition model M3 to perform long-time pattern recognition on the audio data S l to obtain the vehicle recognition result R l and its corresponding long-time classification probability P l ; the audio data S l is the first l frames of audio data collected in step 3), where l ∈ {2, 3,..., N}; the vehicle recognition result R l is the output of the incrementally updated vehicle recognition model M3, that is, the vehicle type with the highest model classification probability; the short-time classification probability P l is the confidence of the incrementally updated vehicle recognition model M3 in the vehicle recognition result R l , and the calculation method is as follows:

[0118]

[0119] where E mel_l is the Mel cepstral coefficient energy, X l [k] is the frequency domain energy of the audio data S l , H m [k] is the parameter of the m-th Mel filter, z i is the probability of the i-th category output by the model, C is the total number of classification categories, and e is the base of the natural logarithm; at the same time, the long-time classification probability threshold P′ lThe dynamic adjustment is as follows:

[0120] P' l = μ hist + k·σ hist

[0121] Judge whether the long - term classification probability P l is greater than or equal to the long - term classification probability threshold P' l , if so, output a vehicle recognition result R l ; otherwise, let l = l + 1, judge whether l is less than or equal to N, if so, repeat step 41); otherwise, save the audio data S N to the difficult sample set D hard and execute step 42);

[0122] 42) Output that the vehicle recognition result is unrecognizable, and judge whether the number of samples in the difficult sample set D hard is equal to the set upper limit K of the number of samples. If so, upload the difficult sample set to the server, clear the locally saved difficult sample set and return to step 2); otherwise, directly return to step 2).

[0123] The specific application ways of the present invention are numerous. The above - mentioned are only the preferred embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements can be made, and these improvements should also be regarded as the protection scope of the present invention.

Claims

1. A vehicle recognition system based on audio data and vibration data, characterized in that, Including: A server and edge devices; among which, The server includes: a model training module and an incremental update module; The model training module is used to train an initial vehicle recognition model using vehicle audio data to obtain a trained vehicle recognition model; The incremental update module is used to incrementally update the trained vehicle recognition model using the vehicle audio data uploaded by the edge device to obtain an incrementally updated vehicle recognition model; The edge device includes: a vibration data acquisition module, an audio data acquisition module, and a vehicle recognition module; The vibration data acquisition module is used to acquire vehicle vibration data and perform vibration energy threshold determination; The audio data acquisition module is used to acquire vehicle audio data and perform audio energy threshold determination; The vehicle recognition module is used to perform pattern recognition on the vehicle audio data using the incrementally updated vehicle recognition model and send the unrecognized vehicle audio data to the incremental update module.

2. A vehicle recognition method based on audio data and vibration data, based on the system described in claim 1, characterized in that, The method steps are as follows: 1) Train an initial vehicle recognition model using vehicle audio data and incrementally update the trained vehicle recognition model using the unrecognized vehicle audio data; 2) Acquire vehicle vibration data and perform vibration energy threshold determination. If it is less than the vibration energy threshold, repeat step 2), otherwise go to step 3); 3) Acquire a single-frame vehicle audio data and perform audio energy threshold determination. If it is less than the audio energy threshold, return to step 2), otherwise, continuously acquire multiple frames of vehicle audio data and go to step 4); 4) Perform pattern recognition on the vehicle audio data acquired in step 3) using the incrementally updated vehicle recognition model in step 1) and output the recognition result, and upload the unrecognized vehicle audio data to the server.

3. The vehicle recognition method based on audio data and vibration data according to claim 2, wherein The specific steps of step 1) include: 11) Train the initial vehicle recognition model M1 using the vehicle audio dataset D1; The vehicle audio dataset D1 includes multiple labeled vehicle audio samples. Each vehicle audio sample is derived from the audio data composed of engine noise and tire noise collected when the vehicle passes through the collection point. The corresponding label is the vehicle type, including cars, buses, and trucks. The expression of the vehicle audio dataset D1 is as follows: D1 = {d1, d2, …, d n} d i = {x i , y i} Among them, n is the total number of dataset samples, and d i is the i-th sample, and x i is the vehicle audio data corresponding to the i-th sample, and y i is the vehicle type corresponding to the i-th sample; The initial vehicle recognition model M1 is a classification model based on a deep convolutional neural network. Its input is vehicle audio data, and the output is the vehicle type with the highest classification probability. The model M1 specifically includes: an input layer, a preprocessing layer, a convolutional layer, a pooling layer, a fully connected layer, and a Softmax output layer. The model M1 is supervised-trained based on the vehicle audio dataset D1. The training process uses the backpropagation algorithm and the gradient descent optimization algorithm to minimize the cross-entropy loss function between the classification output and the true label. The trained vehicle recognition model M2 can realize the mapping from vehicle audio data to vehicle type. The expression is as follows: Among them, is the input space, is the output space, x is a specific vehicle audio data in the input space, and θ is the set of trainable parameters of the model; 12) Incrementally update the trained vehicle recognition model M2 to obtain the incrementally updated vehicle recognition model M3. Specifically, upon receiving the difficult sample set D composed of unrecognizable vehicle audio data sent by the edge device hard ={S (1) , S (2) , …, S (K)}, where K is the total number of samples in the difficult sample set, and S (i) is the i-th sample, and each sample is vehicle audio data; annotate S (i) to obtain the corresponding vehicle type y i , and construct a new sample as Then perform federated aggregation to update the trained vehicle recognition model M2 as follows: Among them, θ new is the set of trainable parameters of the vehicle recognition model M3 after incremental update, and θ old is the set of trainable parameters of the trained vehicle recognition model M2. η is the learning rate, is the gradient, and L is the cross-entropy loss function; The edge device obtains the parameter set θ new , and adopts a smoothing update strategy to prevent the vehicle recognition model M3 from oscillating after incremental update, ensuring that the model retains its original feature expression ability while adapting to new data. The expression of the smoothing update strategy is as follows: θ new = λθ new +(1 - λ)θ old Among them, λ is the weight coefficient, and λ ∈ [0, 1].

4. The vehicle recognition method based on audio data and vibration data according to claim 2, characterized in that The specific steps of step 2) include: Acquire vehicle vibration data to obtain vibration data of E1×T1, where E1 and T1 respectively represent the vibration energy and duration of the vibration data. The calculation method of E1 is as follows: Wherein, a[n] is the discrete vibration data collected by the vibration sensor, and L1 is the number of vibration sampling points within the duration T1; determine the average vibration energy whether it is less than the preset average vibration energy threshold E′1, or whether the vibration duration T1 is less than the vibration duration threshold T′1, If so, repeat step 2); otherwise, proceed to step 3).

5. The vehicle recognition method based on audio data and vibration data according to claim 2, wherein The specific steps of step 3) include: Collect single-frame vehicle audio data to obtain audio data S1 of E2×T2, where E2 and T2 respectively represent the audio energy and frame length of the single-frame audio data. The calculation method of E2 is as follows: Among them, x[n] is the discrete audio data collected by the microphone, and L2 is the number of sampling points of each frame of audio data; judge the average audio energy whether it is less than the preset average audio energy threshold E2 ′ , if so, return to step 2); otherwise, continuously collect a total of N frames of vehicle audio data to obtain E N ×T N of the audio data S N , where E N represents the total energy of S N , T N represents the total duration of S N , T N = N * T2, and enter step 4).

6. The vehicle recognition method based on audio data and vibration data according to claim 2, characterized in that, Step 4) specifically includes: Short-time mode recognition: Use the incrementally updated vehicle recognition model M3 to perform mode recognition on the single-frame vehicle audio data collected in step 3). If the short-time classification probability is greater than or equal to the short-time classification probability threshold, output the recognition result; otherwise, perform long-time mode recognition; Long-time mode recognition: Use the updated vehicle recognition model M3 to perform mode recognition on the multi-frame vehicle audio data collected in step 3). If the long-time classification probability is greater than or equal to the long-time classification probability threshold, output the recognition result; otherwise, increase the number of audio data frames and repeat the long-time mode recognition; when the number of repetitions reaches the upper limit, save the audio data to the difficult sample set; when the number of samples in the difficult sample set reaches the upper limit, upload the difficult sample set to the server.

7. The vehicle recognition method based on audio data and vibration data according to claim 6, characterized in that The short-time mode recognition specifically includes: Use the incrementally updated vehicle recognition model M3 to perform short-time mode recognition on the audio data S1 to obtain the vehicle recognition result R1 and its corresponding short-time classification probability P1; the vehicle recognition result R1 is the output of the incrementally updated vehicle recognition model M3, that is, the vehicle type with the highest model classification probability; the short-time classification probability P1 is the confidence of the incrementally updated vehicle recognition model M3 in the recognition result R1, and the calculation formula is as follows: Among them, E mel_1 is the energy of Mel cepstral coefficients, X[k] is the frequency-domain energy of audio data S1, H m [k] is the parameter of the m-th Mel filter, M is the total number of Mel filters, z i is the probability of the i-th category output by the model, C is the total number of classification categories, and e is the base of the natural logarithm; at the same time, the dynamic adjustment of the short-time classification probability threshold P′1 is as follows: P′1 = μ hist + k·σ hist where k is the standard deviation adjustment hyperparameter, k ∈ (0, 1), μ hist and σ his are the moving window mean and standard deviation based on historical classification probabilities, respectively, as follows: where W is the window length, is the recognition probability at the t-th time in history; if the short-term classification probability P1 is greater than or equal to the short-term classification probability threshold P1 ′ , then output a vehicle recognition result R1 once, otherwise perform long-term pattern recognition.

8. The vehicle recognition method based on audio data and vibration data according to claim 7, characterized in that, The long-time mode recognition specifically includes: 41) Use the vehicle recognition model M3 after incremental update to perform long-term pattern recognition on the audio data S l to obtain the vehicle recognition result R l and its corresponding long-term classification probability P l ; the audio data S l is the first l frames of audio data collected in step 3), where l ∈ {2, 3, …, N}; the vehicle recognition result R l is the output of the vehicle recognition model M3 after incremental update, that is, the vehicle type with the highest model classification probability; the short-term classification probability P l is the confidence of the vehicle recognition result R l by the vehicle recognition model M3 after incremental update, and the calculation method is as follows: Among them, E mel_l is the energy of Mel cepstral coefficients, X l [k] is the frequency-domain energy of audio data S l , H m [k] is the parameter of the m-th Mel filter, z i is the probability of the i-th category output by the model, C is the total number of classification categories, and e is the base of the natural logarithm; at the same time, the dynamic adjustment of the long-term classification probability threshold P′ l is as follows: P′ l = μ hist + k·σ hist Judge the long-term classification probability P l Is it greater than or equal to the long-term classification probability threshold P' l If so, output the vehicle recognition result R once l ; Otherwise, let l = l + 1, judge whether l is less than or equal to N. If so, repeat step 41); Otherwise, save the audio data S N To the difficult sample set D hard And execute step 42); 42) Output that the vehicle recognition result is unrecognizable, and judge whether the number of samples in the difficult sample set D hard is equal to the set upper limit K of the number of samples. If so, upload the difficult sample set to the server, clear the locally saved difficult sample set, and return to step 2); otherwise, directly return to step 2).

Images