CNN-TCN-RNN-based vehicle vibration response signal prediction method
Through the CNN-TCN-RNN method combined with road surface images and vibration signals, efficient and accurate prediction of vehicle vibration response signals is achieved, solving the problem of single data source and simplified distortion in traditional methods, and improving prediction accuracy.
Patent Information
- Application Number
- CN202510456104.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-11
- Publication Date
- 2025-08-01
AI Technical Summary
In traditional vehicle vibration response signal prediction, the data source modality is single and the model simplified distortion is caused by insufficient prediction accuracy under complex operating conditions, making it difficult to take into account the structural nonlinear characteristics and calculation time.
The CNN-TCN-RNN method is adopted to collect road surface image features through vehicle-mounted cameras, combine the vibration signals of the acceleration sensor, and extract features using convolutional neural network (CNN) and time convolutional network (TCN) and predictions are made by recurrent neural network (RNN) after fusing multimodal data.
It improves the prediction accuracy of vehicle vibration response signals, realizes efficient and accurate prediction under complex and variable road excitation, and supports applications such as vehicle active control and fault warning.
Smart Images
Figure CN120408497A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision and signal processing, and mainly relates to a method for predicting vehicle vibration response signals. Background Art
[0002] At present, there are two major limitations in the field of vehicle vibration response signal prediction. On the one hand, the traditional prediction data source has a single modality, relying only on vibration signals or idealized road excitation data, resulting in incomplete extraction of excitation features; on the other hand, the traditional prediction model is simplified and distorted, usually being a vehicle system dynamics model or a finite element structure model, which is difficult to take into account both structural nonlinear characteristics and calculation time. Thus, it restricts the prediction accuracy under complex driving conditions of vehicles.
[0003] Regarding the application of vehicle vibration response signal prediction, the development in many fields is relatively mature. At present, vehicle vibration response signal prediction has broken through the scope of single fault diagnosis and is widely applied in fields such as suspension structure optimization, autonomous driving comfort control, and new energy vehicle energy management. With the integration of multi-modal perception and edge computing technologies, its application scenarios are extending towards intelligent transportation networks, unmanned equipment clusters, etc., becoming the core support technology of intelligent transportation systems. Therefore, to accurately predict the vehicle vibration response under complex and variable road excitations in real time, it is necessary to establish an efficient and accurate vehicle vibration response signal prediction classification method, effectively utilize multi-modal data sources, improve the prediction accuracy of vehicle vibration response signals, provide high-precision dynamic prediction data for scenarios such as vehicle active control and fault warning, and ultimately achieve full-road-condition adaptive performance optimization. Summary of the Invention
[0004] Aiming at the problems existing in the above-mentioned prior art, the technical problem to be solved by the present invention is to provide a vehicle vibration response signal prediction method based on CNN-TCN-RNN, which uses a convolutional neural network (CNN) to extract road surface image features, uses a temporal convolutional network (TCN) to extract vibration signal features, and finally processes multi-modal features through a recurrent neural network (RNN) to achieve accurate prediction of vehicle vibration.
[0005] The implementation steps of the technical solution are as follows:
[0006] (1) Collect and align multi-modal data:
[0007] Collect the front road surface RGB images through an on-vehicle camera at a frequency of f img to obtain the original image sequence p(t). Collect vibration signals through an acceleration sensor at a frequency of f vibCollect the time-series signal of the vehicle vibration response at a certain frequency to obtain the original vibration signal q(t). Segment the signal, with each segment called a frame, to align it with the image frame rate. Finally, obtain the road surface image input sequence I(t) and the vibration signal input sequence V(t). That is, for each image acquisition moment, there is:
[0008] I(t) = p(t)
[0009]
[0010] In the formula, f img represents the sampling frequency of the on-vehicle camera, f vib represents the sampling frequency of the acceleration sensor, N represents the number of samples included in each frame during the segmentation of the original vibration signal, p(t) represents the original image sequence, q(t) represents the original vibration signal, I(t) represents the road surface image input sequence, and V(t) represents the vibration signal input sequence.
[0011] (2) Construct a vehicle vibration excitation and response triple dataset:
[0012] For each sample point i, extract the current road surface image frame I(i), the historical vibration signal H(i), and the current vibration signal frame V(i) from the road surface image input sequence I(t) and the vibration signal input sequence V(t) to form a triple, as follows:
[0013] D = {(I, H, V)} = {(I(i), H(i), V(i))|i = 1, 2,..., M}
[0014] H(i) = [V(i - n), V(i - n + 1),..., V(i - 1)]
[0015] In the formula, D represents the vehicle vibration excitation and response triple dataset, M represents the number of sample points in the dataset, and n represents the number of signal frames included in the historical vibration signal.
[0016] (3) Construct a convolutional neural network (CNN) to extract road surface features:
[0017] Input the three-channel image I(i), preprocess the image, scale the image to 224×224 pixels, normalize the pixel values to the range [0, 1], and standardize it according to the mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] of the ImageNet dataset. At the same time, perform image enhancement operations such as random cropping, horizontal flipping, and slight perturbations of brightness / contrast on the image. Use kernel convolution on the preprocessed image to extract the road surface image features, and finally obtain its feature matrix, that is:
[0018]
[0019] In the formula, f represents the activation function, b1 represents the bias term of the convolution, s and p represent the stride and padding of the convolution, Y represents the output feature matrix, K represents the convolution kernel, I represents the preprocessed input image matrix, Y(i,j) represents the element at the i-th row and j-th column of the feature matrix, K(m,n,c) represents the element at the m-th row, n-th column and c-th channel of the convolution kernel, and I(is+m-p,js+n-p,c) represents the element at the (is+m-p)-th row, (js+n-p)-th column and c-th channel of the image matrix.
[0020] (4) Construct a Temporal Convolutional Network (TCN) to extract vibration features:
[0021] Input the historical vibration time series signal H(i), preprocess the vibration signal, and improve the model generalization through noise addition and time warping operations. Apply causal convolution to the preprocessed historical vibration time series signal to extract the vibration signal features, and finally obtain its feature matrix, that is:
[0022]
[0023] In the formula, f represents the activation function, b2 represents the bias term of the causal convolution, y represents the output feature sequence, w represents the causal convolution kernel, x represents the input signal sequence, k represents the length of the causal convolution kernel, y(t) represents the i-th element of the output feature sequence, w(m) represents the m-th element of the causal convolution kernel, and x(t-m) represents the (t-m)-th element of the input signal sequence.
[0024] (5) Fuse multi-modal features:
[0025] Fuse the image feature matrix extracted by the Convolutional Neural Network (CNN) and the signal feature sequence extracted by the Temporal Convolutional Network (TCN). Compress the image feature matrix into a one-dimensional sequence through average value calculation, and finally splice to obtain the fused feature sequence, that is:
[0026]
[0027] G = [Y'; y]
[0028] In the formula, Y' represents the compressed one-dimensional sequence of image features, Y represents the image feature matrix extracted by the Convolutional Neural Network (CNN), y represents the signal feature sequence extracted by the Temporal Convolutional Network (TCN), and G represents the fused feature sequence.
[0029] (6) Construct a Recurrent Neural Network (RNN) to predict vibration signals:
[0030] Take the fused feature sequence G as the input of the recurrent neural network (RNN). The RNN unit updates the current hidden state at each time step based on the current input and the hidden state of the previous time step, and finally obtains the prediction result through a linear transformation based on the hidden state of the last time step, that is:
[0031] h(t) = f(W hh h(t-1) + W gh G(t) + b h )
[0032]
[0033] In the formula, f represents the activation function, h(t) represents the hidden state at the t-th time step, G(t) represents the input at the t-th time step, W hh represents the weight matrix from the hidden state to the hidden state, W gh represents the weight matrix from the input to the hidden state, b h represents the bias vector of the hidden state, represents the prediction result, W hy represents the weight matrix from the hidden state to the output, T represents the sequence length, b y represents the bias vector of the output.
[0034] (7) Determine the loss function for signal prediction:
[0035] Use the mean squared error (MSE) to evaluate the difference between the predicted value and the true value, that is:
[0036]
[0037] In the formula, M represents the number of samples, represents the predicted value of the i-th sample point, and V(i) represents the true value of the i-th sample point. Through training and iteration, finally make the mean squared error (MSE) optimal.
[0038] Advantages of the present invention over the prior art:
[0039] (1) The present invention combines the road surface image (visual modality) with the vibration signal (time series modality), breaks through the limitations of traditional single-modal prediction in the field of vehicle vibration response signal prediction, and can effectively improve the prediction accuracy of vehicle vibration response signals.
[0040] (2) The present invention applies the advantage that deep learning algorithms can use linear learning algorithms to complete non - linear learning algorithms to the regression prediction of vehicle vibration response signals, and achieves a high prediction accuracy by integrating convolutional neural network (CNN), temporal convolutional network (TCN) and recurrent neural network (RNN). This shows that when the present invention predicts vehicle vibration response signals, a good prediction effect can be achieved. Description of the Drawings
[0041] To better understand the present invention, the following further explains with reference to the accompanying drawings.
[0042] Figure 1 is the flow chart of the steps for establishing a vehicle vibration response signal prediction method based on CNN - TCN - RNN;
[0043] Figure 2 is the algorithm flow chart of the vehicle vibration response signal prediction method based on CNN - TCN - RNN;
[0044] Figure 3 is the structural schematic diagram of convolutional neural network (CNN);
[0045] Figure 4 is the structural schematic diagram of temporal convolutional network (TCN);
[0046] Figure 5 is the structural schematic diagram of recurrent neural network (RNN);
[0047] Figure 6 is the result of predicting four groups of vehicle vibration response signals using the present invention. Detailed Embodiments
[0048] The following further clarifies the present invention with reference to the accompanying drawings. The following specific embodiments are only used to illustrate the present invention and not to limit the scope of the present invention.
[0049] The overall process of the vehicle vibration response signal prediction method based on CNN - TCN - RNN provided by the present invention is as Figure 1 shown, and the specific steps are as follows:
[0050] (1) Collect and align multi - modal data:
[0051] Collect the RGB images of the road surface ahead at a frequency of 50Hz through an on - vehicle camera to obtain the original image sequence p(t). Collect the vehicle vibration response time - series signal at a frequency of 1kHz through an acceleration sensor to obtain the original vibration signal q(t). Segment the signal, and each segment is called a frame, and align it with the image frame rate. Finally, obtain the road surface image input sequence I(t) and the vibration signal input sequence V(t). That is, for each image acquisition moment, there is:
[0052] I(t) = p(t)
[0053]
[0054] where f img represents the sampling frequency of the vehicle-mounted camera, f vib represents the sampling frequency of the acceleration sensor, N represents the number of samples contained in each frame when the original vibration signal is segmented, p(t) represents the original image sequence, q(t) represents the original vibration signal, I(t) represents the road surface image input sequence, and V(t) represents the vibration signal input sequence.
[0055] (2) Construct a vehicle vibration excitation and response triple dataset:
[0056] For each sample point i, extract the current road surface image frame I(i), historical vibration signal H(i), and current vibration signal frame V(i) from the road surface image input sequence I(t) and vibration signal input sequence V(t) to form a triple, as follows:
[0057] D = {(I, H, V)} = {(I(i), H(i), V(i))|i = 1, 2,..., M} = {(I(i), H(i), V(i))|i = 1, 2,..., 360000}
[0058] H(i) = [V(i - n), V(i - n + 1),..., V(i - 1)] = [V(i - 30), V(i - 30 + 1),..., V(i - 1)]
[0059] where D represents the vehicle vibration excitation and response triple dataset, M represents the number of sample points in the dataset, and n represents the number of signal frames included in the historical vibration signal. In this implementation case, the selected dataset has a total of 100000 groups of samples, 70000 samples are used for training, and 30000 samples are used for testing.
[0060] (3) Construct a convolutional neural network (CNN) to extract road surface features:
[0061] Input the three-channel image I(i), preprocess the image, scale the image to 224×224 pixels, normalize the pixel values to the range [0, 1], and standardize according to the mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] of the ImageNet dataset. At the same time, perform image enhancement operations such as random cropping, horizontal flipping, and slight perturbations of brightness / contrast on the image. Use kernel convolution on the preprocessed image to extract road surface image features, and finally obtain its feature matrix, that is:
[0062]
[0063] In the formula, f represents the activation function, b1 represents the bias term of the convolution, s and p represent the convolution stride and padding, Y represents the output feature matrix, K represents the convolution kernel, I represents the preprocessed input image matrix, Y(i, j) represents the element at the i-th row and j-th column of the feature matrix, K(m, n, c) represents the element at the m-th row, n-th column and c-th channel of the convolution kernel, and I(is + m - p, js + n - p, c) represents the element at the (is + m - p)-th row and (js + n - p)-th column and c-th channel of the image matrix.
[0064] (4) Construct a Temporal Convolutional Network (TCN) to extract vibration features:
[0065] Input the historical vibration time series signal H(i), preprocess the vibration signal, and improve the model generalization through noise addition and time warping operations. Apply causal convolution to the preprocessed historical vibration time series signal to extract the vibration signal features, and finally obtain its feature matrix, that is:
[0066]
[0067] In the formula, f represents the activation function, b2 represents the bias term of the causal convolution, y represents the output feature sequence, w represents the causal convolution kernel, x represents the input signal sequence, k represents the length of the causal convolution kernel, y(t) represents the i-th element of the output feature sequence, w(m) represents the m-th element of the causal convolution kernel, and x(t - m) represents the (t - m)-th element of the input signal sequence.
[0068] (5) Fuse multi-modal features:
[0069] Fuse the image feature matrix extracted by the Convolutional Neural Network (CNN) and the signal feature sequence extracted by the Temporal Convolutional Network (TCN). Compress the image feature matrix into a one-dimensional sequence through average value calculation, and finally splice to obtain the fused feature sequence, that is:
[0070]
[0071] G = [Y'; y]
[0072] In the formula, Y' represents the compressed one-dimensional sequence of image features, Y represents the image feature matrix extracted by the Convolutional Neural Network (CNN), y represents the signal feature sequence extracted by the Temporal Convolutional Network (TCN), and G represents the fused feature sequence.
[0073] (6) Construct a Recurrent Neural Network (RNN) to predict vibration signals:
[0074] Use the fused feature sequence G as the input of a recurrent neural network (RNN). The RNN unit updates the current hidden state at each time step based on the current input and the hidden state of the previous time step, and finally obtains the prediction result through a linear transformation based on the hidden state of the last time step, that is:
[0075] h(t) = f(W hh h(t - 1)+W gh G(t)+b h )
[0076]
[0077] where f represents the activation function, h(t) represents the hidden state at the t-th time step, G(t) represents the input at the t-th time step, W hh represents the weight matrix from the hidden state to the hidden state, W gh represents the weight matrix from the input to the hidden state, b h represents the bias vector of the hidden state, represents the prediction result, W hy represents the weight matrix from the hidden state to the output, T represents the sequence length, b y represents the bias vector of the output.
[0078] (7) Determine the loss function for signal prediction:
[0079] Use the mean squared error (MSE) to evaluate the difference between the predicted value and the true value, that is:
[0080]
[0081] where M represents the number of samples, represents the predicted value of the i-th sample point, and V(i) represents the true value of the i-th sample point. Through training and iteration, the mean squared error (MSE) is finally optimized.
[0082] To verify the accuracy of the vehicle vibration response signal prediction of the present invention, four groups of vibration signal prediction experiments were carried out on the present invention, and the experimental results are as Figure 6 shown. It can be Figure 6 seen that the accuracy rate of the vehicle vibration response prediction method established by the present invention for predicting the vehicle vibration response signal remains above 95%. It can achieve a relatively high accuracy rate on the basis of ensuring stability, and the prediction effect is good. This shows that the vehicle vibration response signal prediction method established by the present invention is effective, provides a better method for establishing an accurate vibration signal prediction model, and has a certain practicality.
Claims
1. A vehicle vibration response signal prediction method based on CNN-TCN-RNN, characterized in that: (1) Collect and align multi-modal data; (2) Construct a vehicle vibration excitation and response triple dataset; (3) Construct a Convolutional Neural Network (CNN) to extract road surface features; (4) Construct a Temporal Convolutional Network (TCN) to extract vibration features; (5) Fuse multi-modal features; (6) Construct a Recurrent Neural Network (RNN) to predict vibration signals; (7) Determine the loss function for signal prediction. Specifically, it includes the following seven steps: Step 1: Collect and align multi-modal data: Collect the RGB images of the road ahead through an in-vehicle camera at a frequency of f img to obtain the original image sequence p(t). Collect the vehicle vibration response time series signal through an acceleration sensor at a frequency of f vib to obtain the original vibration signal q(t). Segment the signal, with each segment called a frame, to align it with the image frame rate, and finally obtain the road surface image input sequence I(t) and the vibration signal input sequence V(t). That is, for each image acquisition moment, there is: I(t) = p(t) where f img represents the sampling frequency of the vehicle-mounted camera, f vib represents the sampling frequency of the acceleration sensor, N represents the number of samples included in each frame during the segmentation of the original vibration signal, p(t) represents the original image sequence, q(t) represents the original vibration signal, I(t) represents the road surface image input sequence, and V(t) represents the vibration signal input sequence; Step 2: Construct a vehicle vibration excitation and response triple dataset: For each sample point i, extract the current road surface image frame I(i), historical vibration signal H(i), and current vibration signal frame V(i) from the road surface image input sequence I(t) and vibration signal input sequence V(t) to form a triple, as follows: D = {(I, H, V)} = {(I(i), H(i), V(i))|i = 1, 2,..., M} H(i) = [V(i - n), V(i - n + 1),..., V(i - 1)] In the formula, D represents the vehicle vibration excitation and response triple dataset, M represents the number of sample points in the dataset, and n represents the number of signal frames included in the historical vibration signal; Step 3: Construct a Convolutional Neural Network (CNN) to extract road surface features: Input the three-channel image I(i), preprocess the image, scale the image to 224×224 pixels, normalize the pixel values to the range of [0, 1], and standardize according to the mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] of the ImageNet dataset. At the same time, perform image enhancement operations such as random cropping, horizontal flipping, and slight perturbations of brightness / contrast on the image. Use kernel convolution on the preprocessed image to extract road surface image features, and finally obtain its feature matrix, that is: In the formula, f represents the activation function, b1 represents the bias term of the convolution, s and p represent the convolution stride and padding, Y represents the output feature matrix, K represents the convolution kernel, I represents the input preprocessed image matrix, Y(i, j) represents the element in the i-th row and j-th column of the feature matrix, K(m, n, c) represents the element in the m-th row, n-th column, and c-th channel of the convolution kernel, and I(is + m - p, js + n - p, c) represents the element in the (is + m - p)-th row and (js + n - p)-th column of the image matrix in the c-th channel; Step 4: Construct a Temporal Convolutional Network (TCN) to extract vibration features: Input the historical vibration time series signal H(i), preprocess the vibration signal, and improve the model generalization through noise addition and time warping operations. Use causal convolution on the preprocessed historical vibration time series signal to extract vibration signal features, and finally obtain its feature matrix, that is: Wherein, f represents the activation function, b2 represents the bias term of the causal convolution, y represents the output feature sequence, w represents the causal convolution kernel, x represents the input signal sequence, k represents the length of the causal convolution kernel, y(t) represents the i-th element of the output feature sequence, w(m) represents the m-th element of the causal convolution kernel, and x(t - m) represents the (t - m)-th element of the input signal sequence; Step Five: Fuse multi-modal features: Fuse the image feature matrix extracted by the convolutional neural network (CNN) and the signal feature sequence extracted by the temporal convolutional network (TCN). Compress the image feature matrix into a one-dimensional sequence through average value calculation, and finally splice to obtain the fused feature sequence, that is: Wherein, Y' represents the one-dimensional sequence of the compressed image features, Y represents the image feature matrix extracted by the convolutional neural network (CNN), y represents the signal feature sequence extracted by the temporal convolutional network (TCN), and G represents the fused feature sequence; Step Six: Construct a recurrent neural network (RNN) to predict vibration signals: Use the fused feature sequence G as the input of the recurrent neural network (RNN). The RNN unit updates the current hidden state according to the current input and the hidden state of the previous time step at each time step. Finally, the prediction result is obtained through a linear transformation based on the hidden state of the last time step, that is: h(t) = f(W hh h(t - 1)+W gh G(t)+b h ) where f represents the activation function, h(t) represents the hidden state at the t-th time step, G(t) represents the input at the t-th time step, W hh represents the weight matrix from the hidden state to the hidden state, W gh represents the weight matrix from the input to the hidden state, b h represents the bias vector of the hidden state, represents the prediction result, W hy represents the weight matrix from the hidden state to the output, T represents the sequence length, b y represents the bias vector of the output; Step Seven: Determine the loss function for signal prediction: Use the mean square error (MSE) to evaluate the difference between the predicted value and the true value, that is: Where M represents the number of samples, represents the predicted value of the i-th sample point, and V(i) represents the true value of the i-th sample point. Through training and iteration, the mean squared error (MSE) is finally optimized.