Vehicle behavior monitoring method and device, electronic equipment and computer storage medium

Abstract

The invention provides a vehicle behavior monitoring method and device, electronic equipment and a computer storage medium. The method comprises the following steps: preprocessing vehicle driving data; the vehicle driving data comprises sensor data and image data; extracting vehicle behavior characteristics in the vehicle driving data; the vehicle behavior characteristics are configured to fuse multi-modal information characteristics; and carrying out classification and anomaly monitoring on the vehicle behavior characteristics through a multi-task deep learning model. According to the method, the sensor data and the image data are extracted to obtain the fused vehicle driving data, and the vehicle behavior characteristics of the vehicle driving data are extracted to obtain the fused vehicle characteristics, so that the vehicle behavior characteristics are classified and abnormal monitoring is performed through the fused vehicle characteristics and the multi-task deep learning model; the accuracy of vehicle behavior classification and anomaly monitoring can be improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle monitoring. Specifically, it relates to a vehicle behavior monitoring method, device, electronic device, and computer storage medium. Background Art

[0002] Traditional vehicle behavior monitoring methods mainly rely on cameras or sensors. As the most commonly used monitoring tool, a camera captures video images to identify information such as vehicle type, speed, and driving trajectory. However, its effect is easily affected by lighting conditions, weather changes (such as rain, fog, and night), camera installation angle, and clarity limitations, resulting in a decrease in data accuracy in some complex environments. Although sensors can, to a certain extent, make up for the deficiencies of cameras in some aspects, such as being unaffected by light and providing more accurate distance and speed measurements, the coverage range of a single sensor is limited. Therefore, whether vehicle behavior is monitored through a camera or through a sensor, the accuracy of the monitored vehicle behavior is relatively low. Summary of the Invention

[0003] In view of this, the purpose of the embodiments of the present application is to provide a vehicle behavior monitoring method, device, electronic device, and computer storage medium, which can improve the accuracy of vehicle behavior monitoring.

[0004] In a first aspect, the embodiments of the present application provide a vehicle behavior monitoring method, including: preprocessing vehicle driving data; where the vehicle driving data includes sensor data and image data; extracting vehicle behavior features from the vehicle driving data; where the vehicle behavior features are configured to fuse multi-modal information features; classifying and abnormally monitoring the vehicle behavior features through a multi-task deep learning model.

[0005] In the above implementation process, by extracting sensor data and image data, fused vehicle driving data can be obtained, and by extracting vehicle behavior features from the vehicle driving data, fused vehicle features can be obtained. Furthermore, through the fused vehicle features and the multi-task deep learning model, the vehicle behavior features can be classified and abnormally monitored, which can improve the accuracy of vehicle behavior classification and abnormal monitoring.

[0006] In an embodiment, the extracting vehicle behavior features from the vehicle driving data includes: extracting sensor data features from the sensor data through a recurrent neural network structure; extracting image data features from the image data through a convolutional neural network; and performing feature fusion on the sensor data features and the image data features by using a weighted summation method to obtain the vehicle behavior features.

[0007] In the above implementation process, the sensor data features in the sensor data are extracted through a recurrent neural network structure, and the image data features in the image data are extracted through a convolutional neural network. The sensor data features and the image data features are fused by weighted summation to obtain vehicle behavior features, which can improve the monitoring accuracy through multi-source data fusion and enhance the accuracy of vehicle behavior monitoring.

[0008] In one embodiment, extracting the sensor data features in the sensor data through the recurrent neural network structure includes: inputting the sensor data at each moment into the recurrent neural network structure to calculate the hidden state at each moment; wherein, a gating mechanism is included in the recurrent neural network structure; taking the hidden state at the final moment as the sensor data features of the sensor data.

[0009] In the above implementation process, by introducing a gating mechanism into the recurrent neural network structure, the processing ability of the recurrent neural network structure for long sequence data can be improved, the efficiency and flexibility of the recurrent neural network structure can be enhanced, and the accuracy of the sensor data features can be increased.

[0010] In one embodiment, extracting the image data features in the image data through the convolutional neural network includes: performing a convolution operation on the image data and the convolution kernels in the convolutional neural network to extract the feature map of the image data; multiplying the attention weights by the feature map to obtain a weighted feature map; processing the weighted feature map through a pooling layer and a fully connected layer to obtain the image data features.

[0011] In the above implementation process, by introducing an attention mechanism into the convolutional neural network, the convolutional neural network can focus on the regions in the image that are more important for vehicle behavior judgment, thereby improving the accuracy of image data feature extraction and enhancing the accuracy of vehicle behavior monitoring.

[0012] In one embodiment, preprocessing the vehicle driving data includes: calculating the mean and standard deviation of the sensor data, and determining whether the sensor data is abnormal according to the mean and the standard deviation; in the case where the sensor data is abnormal, correcting the abnormal sensor data; performing normalization processing on the corrected sensor data; normalizing the image data to a specific mean and standard deviation; and performing data fusion on the normalized sensor data and the normalized image data.

[0013] In the above implementation process, before extracting the vehicle behavior features, preprocessing methods such as cleaning, correcting, and normalizing the vehicle driving data can reduce the abnormal data in the vehicle driving data, improve the accuracy of the vehicle driving data, and at the same time, can also reduce the processing difficulty of the vehicle driving data.

[0014] In one embodiment, the classification and anomaly monitoring of the vehicle behavior characteristics by the multi-task deep learning model includes: classifying the vehicle behavior characteristics through a fully-connected neural network structure; wherein, the activation function of the hidden layer of the fully-connected neural network structure is the Relu function, the output layer is the softmax function, and the loss function is the cross-entropy loss function; determining the common characteristics of vehicle behavior through the fully-connected neural network structure, and determining the anomaly score through the fully-connected layer; wherein, the loss function for anomaly monitoring is the mean square error loss function.

[0015] In the above implementation process, by using a multi-task deep learning model to classify the vehicle behavior characteristics and determine the anomaly score, the feature differences and correlations of the vehicle in different behavior patterns can be learned, thereby improving the accuracy and reliability of vehicle behavior analysis and anomaly detection.

[0016] In one embodiment, after the classification and anomaly monitoring of the vehicle behavior characteristics by the multi-task deep learning model, the method further includes: determining the warning level according to the classification category and / or anomaly score corresponding to the vehicle behavior characteristics; determining the warning push range and method according to the warning level and the current position of the vehicle.

[0017] In the above implementation process, by determining the warning level according to the classification category and / or anomaly score, and determining the warning push range and method according to the warning level and the current position of the vehicle, the limitations of the existing vehicle behavior detection and warning methods can be solved, and the accuracy, scientificity and flexibility of the warning can be improved.

[0018] In a second aspect, an embodiment of the present application further provides a vehicle behavior monitoring device, including: a preprocessing module for preprocessing vehicle driving data; wherein, the vehicle driving data includes sensor data and image data; an extraction module for extracting vehicle behavior characteristics from the vehicle driving data; wherein, the vehicle behavior characteristics are configured to fuse multi-modal information characteristics; a monitoring module for classifying and anomaly monitoring the vehicle behavior characteristics through a multi-task deep learning model.

[0019] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor and a memory, the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the machine-readable instructions are executed by the processor to execute the method steps in the first aspect, or any possible implementation manner of the first aspect.

[0020] Fourthly, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the vehicle behavior monitoring method in the above first aspect or any possible implementation manner of the first aspect.

[0021] To make the above objects, features, and advantages of the present application more obvious and understandable, specific embodiments are hereinafter given and described in detail in conjunction with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 A block diagram of the electronic device provided by the embodiment of the present application;

[0024] Figure 2 A flowchart of the vehicle behavior monitoring method provided by the embodiment of the present application;

[0025] Figure 3 A flowchart of a specific embodiment of the vehicle behavior monitoring method provided by the embodiment of the present application;

[0026] Figure 4 A schematic diagram of the functional modules of the vehicle behavior monitoring device provided by the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application.

[0028] It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, the terms "first", "second", etc. are only used for descriptive distinction and cannot be understood as indicating or implying relative importance.

[0029] With the continuous increase in the number of automobiles, road traffic safety issues have become increasingly prominent. Abnormal behaviors of vehicles, such as speeding, reverse driving, illegal lane changing, etc., are important factors leading to traffic accidents. Traditional vehicle behavior monitoring methods mainly rely on cameras and sensors, but these methods have certain limitations. The coverage of traffic cameras is limited, and their performance will be affected under adverse weather conditions; the accuracy and reliability of sensor data also need to be improved. In addition, it is difficult for traditional methods to analyze and warn vehicle behaviors in real time and accurately.

[0030] Deep learning technology has achieved great success in the fields of image recognition, speech recognition, etc. Its powerful feature learning ability and pattern recognition ability can provide strong support for vehicle behavior monitoring and safety warning.

[0031] In view of this, this application proposes a vehicle behavior monitoring method. By extracting sensor data and image data, fused vehicle driving data can be obtained. And by extracting the vehicle behavior characteristics of the vehicle driving data, fused vehicle characteristics can be obtained. Furthermore, through the fused vehicle characteristics and a multi-task deep learning model, the vehicle behavior characteristics can be classified and abnormally monitored, which can improve the accuracy of vehicle behavior classification and abnormal monitoring.

[0032] To facilitate the understanding of this embodiment, the electronic device that executes the vehicle behavior monitoring method disclosed in this application embodiment will be introduced in detail first.

[0033] As Figure 1 shown, it is a block diagram of the electronic device. The electronic device 100 may include a memory 111 and a processor 113. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only for illustration and does not limit the structure of the electronic device 100. For example, the electronic device 100 may also include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0034] The above-mentioned memory 111 and processor 113 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components may be electrically connected to each other through one or more communication buses or signal lines. The above-mentioned processor 113 is used to execute the executable module stored in the memory.

[0035] Among them, the memory 111 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc. Among them, the memory 111 is used to store a program, and after receiving an execution instruction, the processor 113 executes the program. The method executed by the electronic device 100 defined by the process disclosed in any embodiment of the embodiments of the present application can be applied to the processor 113 or implemented by the processor 113.

[0036] The above-mentioned processor 113 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 113 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0037] The electronic device 100 in this embodiment can be used to execute each step in the various methods provided in the embodiments of the present application. The implementation process of the vehicle behavior monitoring method will be described in detail through several embodiments below.

[0038] Please refer to Figure 2 which is a flowchart of the vehicle behavior monitoring method provided by the embodiments of the present application. The following will elaborate in detail on the Figure 2 specific process shown.

[0039] Step 201, preprocess the vehicle driving data.

[0040] Among them, the vehicle driving data includes sensor data and image data.

[0041] The sensor data here refers to the data obtained through one or more sensors of the vehicle itself. For example, speed sensors, acceleration sensors, steering wheel angle sensors, etc. The sensors used to obtain the sensor data can be selected according to the actual situation.

[0042] In one embodiment, the vehicle sensors collect data such as speed, acceleration, and steering wheel angle at a high frequency (for example, every 0.1 second).

[0043] The above-mentioned image data refers to the data obtained through image acquisition devices. For example, cameras, pan-tilt heads, sky eyes, etc. The image acquisition devices used to obtain the image data can be selected according to the actual situation.

[0044] In one embodiment, the image acquisition device acquires images of the front, rear, and surrounding environment of the vehicle at a rate of n frames per second (n is determined according to the performance of the camera).

[0045] In addition, information about surrounding vehicles and road facilities in the vehicle networking environment can also be collected through wireless communication technology.

[0046] It should be understood that after obtaining the vehicle driving data, in order to reduce the difficulty of data processing and improve the accuracy and efficiency of data processing, the vehicle driving data can be preprocessed first, and the vehicle behavior characteristics can be classified and abnormal monitored according to the preprocessed vehicle driving data.

[0047] The preprocessing here can include anomaly detection, data cleaning, normalization processing, standardization processing, data fusion, etc. This preprocessing method can be selected according to the actual situation.

[0048] Step 202, extract the vehicle behavior characteristics from the vehicle driving data.

[0049] Among them, the vehicle behavior characteristics are configured to fuse multi-modal information characteristics.

[0050] The vehicle behavior characteristics here can be extracted through one or more neural networks. For example, recurrent neural networks, convolutional neural networks, etc. The extraction method of the vehicle behavior characteristics can be adjusted according to the actual situation.

[0051] The extraction of the vehicle behavior characteristics here can make full use of the multi-modal information (including sensor numerical data and image data, etc.) in the vehicle driving data.

[0052] Suppose the vehicle driving data is D = {d1, d2, …, d T}. Among them, d t represents the data collected at time t, including sensor data (such as speed, acceleration, etc.) and image data. Define the feature extraction function as: F, whose goal is to map the input data D to the feature vector Γ = F(D).

[0053] Step 203, classify and monitor anomalies in vehicle behavior features through a multi-task deep learning model.

[0054] The multi-task deep learning model here can be used to learn the feature differences and correlations of vehicles in different behavior patterns to simultaneously classify vehicle behavior and detect anomalies.

[0055] Among them, the input of the multi-task deep learning model can be a vehicle behavior feature vector, and the output can include vehicle behavior category prediction and anomaly behavior scores.

[0056] Optionally, the vehicle behavior category prediction can include normal driving, speeding, hard braking, reverse driving, illegal lane change, etc., and the specific types of the vehicle behavior category prediction can be selected according to the actual situation.

[0057] The above anomaly behavior score is used to measure the degree of vehicle behavior anomaly.

[0058] In the above implementation process, by extracting sensor data and image data, fused vehicle driving data can be obtained, and by extracting the vehicle behavior features of the vehicle driving data, fused vehicle features can be obtained. Furthermore, by using the fused vehicle features and the multi-task deep learning model to classify and monitor vehicle behavior features, the accuracy of vehicle behavior classification and anomaly monitoring can be improved.

[0059] In a possible implementation manner, step 202 includes: extracting sensor data features in sensor data through a recurrent neural network structure; extracting image data features in image data through a convolutional neural network; and performing feature fusion on the sensor data features and the image data features by using a weighted summation method to obtain vehicle behavior features.

[0060] In one embodiment, when extracting sensor data, an improved recurrent neural network structure can be used to extract features from the preprocessed sensor data (such as speed, acceleration, steering wheel angle, etc.).

[0061] The recurrent neural network structure here is an artificial neural network structure specifically used to process sequential data. It introduces recurrent connections in the network so that information can be circulated between neurons, thereby being able to capture and utilize the temporal information and context relationships in sequential data.

[0062] Among them, the recurrent neural network structure may include an input layer, a hidden layer, and an output layer. The input layer is used to receive the input of sequence data, such as text, speech signals, etc., and transfer it to the hidden layer. The hidden layer is used to receive the input at the current moment and its own state at the previous moment (i.e., the hidden state), enabling the network to maintain a "memory" state, which contains past information and helps to understand the context information in the sequence. The output layer is used to generate corresponding outputs according to the task requirements, such as classification results, generated text, or predicted values, etc.

[0063] The above-mentioned convolutional neural network is a deep learning model specially designed for processing data with grid structures (such as images, audio). It automatically extracts features in the data through components such as convolutional layers, pooling layers, and fully connected layers, without the need to manually design feature extractors.

[0064] Among them, the convolutional neural network includes a convolutional layer, a pooling layer, and a fully connected layer. The convolutional layer is used to slide the convolutional kernel on the input data and perform multiplication and addition operations at each position to extract features. The pooling layer is used to reduce the dimension of the feature map (i.e., width and height), while retaining important features. The fully connected layer is used to further process the extracted features and output the final prediction result.

[0065] It can be understood that after extracting the sensor data features and image data features, it is necessary to fuse the sensor data feature vector and the image data feature vector to obtain the final vehicle behavior feature vector.

[0066] Here, a fusion method based on weighted summation can be used to fuse the sensor data features and image data features. That is, Γ = αΓ s + βΓ i . Among them, α and β are weight coefficients determined according to the data importance, and α + β = 1.

[0067] In one embodiment, by analyzing a large amount of labeled data (such as through methods like principal component analysis), the contribution degrees of sensor data and image data to vehicle behavior judgment can be determined, so as to reasonably set the values of α and β. For example, in a scenario where the vehicle is driving relatively smoothly, the sensor data may be more important, then α can be set to 0.7 and β to 0.3. In complex traffic scenarios (such as intersections), the importance of image data may increase, then α can be adjusted to 0.4 and β to 0.6. Through the above fusion method, the advantages of multi-modal data can be fully utilized, improving the accuracy and effectiveness of vehicle behavior feature extraction and providing more representative feature information for subsequent vehicle behavior analysis and anomaly detection.

[0068] In the above implementation process, the sensor data features in the sensor data are extracted through a recurrent neural network structure, and the image data features in the image data are extracted through a convolutional neural network. The sensor data features and the image data features are feature fused by weighted summation to obtain vehicle behavior features, which can improve the monitoring accuracy through multi-source data fusion and enhance the accuracy of vehicle behavior monitoring.

[0069] In a possible implementation, extracting the sensor data features in the sensor data through a recurrent neural network structure includes: inputting the sensor data at each moment into the recurrent neural network structure to calculate the hidden state at each moment; using the hidden state at the final moment as the sensor data features of the sensor data.

[0070] In an embodiment, when extracting sensor data, an improved recurrent neural network structure can be used to extract features from the preprocessed sensor data (such as speed, acceleration, steering wheel angle, etc.).

[0071] Exemplarily, assume that the input of the recurrent neural network structure at time t is: x t = [v(t), a(t), θ(t)], and the hidden state is h t , then the update formula is: h t = tanh(W ih x t + W hh h t-1 + b h ).

[0072] Among them, W ih and W hh are weight matrices, b h is a bias vector, and tanh is an activation function.

[0073] Through calculations for multiple time steps, the hidden state h T at the final moment is used as the feature vector Γ of the sensor data s = h T .

[0074] Among them, the recurrent neural network structure includes a gating mechanism.

[0075] It can be understood that in order to improve the processing ability of the recurrent neural network structure for long sequence data, a gating mechanism is introduced into the recurrent neural network structure. For example, the gating idea in long short-term memory networks, gated recurrent units, etc.

[0076] Taking LSTM as an example, the update formulas for its cell state and hidden state can be:

[0077] i t = σ(Wxi x t +W hi h t-1 +W ci c t-1 +b i );

[0078] f t =σ(W xf x t +W hf h t-1 +W cf c t-1 +b fc );

[0079] c t =f t Θc t-1 +i t Θh t-1 +W co c t +b o ;

[0080] h t =o t Θtanh(c t );

[0081] Wherein, σ is the sigmoid activation function, Θ represents element-wise multiplication, W is the weight, b is the bias parameter, c t is the cell state at time t, h t is the hidden state at time t, c t-1 is the cell state at time t-1, h t-1 is the hidden state at time t-1, f t is the output of the forget gate, i t is the output of the input gate, o t is the output of the output gate, W xf , W hf and W cf are respectively the weight matrices of the forget gate, b fc is the bias vector of the forget gate, W xi , W hi and W ci are respectively the weight matrices of the input gate, b i is the bias vector of the input gate.

[0082] In the above implementation process, by introducing a gating mechanism into the recurrent neural network structure, the processing ability of the recurrent neural network structure for long sequence data can be improved, the efficiency and flexibility of the recurrent neural network structure can be improved, and the accuracy of sensor data features can be improved.

[0083] In a possible implementation, image data features are extracted from image data through a convolutional neural network, including: performing a convolution operation on the image data and a convolution kernel in the convolutional neural network to extract a feature map of the image data; multiplying the attention weight by the feature map to obtain a weighted feature map; and processing the weighted feature map through a pooling layer and a fully connected layer to obtain image data features.

[0084] Here, the convolutional neural network is used to optimize the image data features.

[0085] Among them, the convolutional layer in the convolutional neural network uses a convolution kernel K ij to perform a convolution operation m×n with the input image I to extract local features of the image.

[0086] Among them, h is the height of the convolution kernel, w is the width of the convolution kernel, K uv is the convolution kernel at the position (u, v), and I i+u,h+v is the pixel value with an offset of (u, v) at the position (i, j).

[0087] In one embodiment, an attention mechanism is introduced into the convolutional neural network. For example, an efficient channel attention module can enable the network to focus on regions in the image that are more important for vehicle behavior judgment.

[0088] Let the feature map after passing through the convolutional layer be F map , and the formula for the efficient channel attention module to calculate the channel attention weight is: w eca =σ(Conv1D(MLP(AvgPool(F map ))))

[0089] Among them, AvgPool represents an average pooling operation, MLP is a multi-layer perceptron, Conv1D is a one-dimensional convolution operation, and σ is an activation function.

[0090] Multiply the attention weight by the feature map to obtain a weighted feature map Then, further process it through a pooling layer and a fully connected layer to obtain the feature vector Γ i of the image data.

[0091] In the above implementation process, by introducing an attention mechanism into the convolutional neural network, the convolutional neural network can focus on regions in the image that are more important for vehicle behavior judgment, thereby improving the accuracy of image data feature extraction and the accuracy of vehicle behavior monitoring.

[0092] In a possible implementation, step 201 includes: calculating the mean and standard deviation of the sensor data, and determining whether the sensor data is abnormal based on the mean and standard deviation; in the case where the sensor data is abnormal, correcting the abnormal sensor data; performing normalization processing on the corrected sensor data; normalizing the image data to a specific mean and standard deviation; and performing data fusion on the normalized sensor data and the normalized image data.

[0093] In one embodiment, the collected data can be cleaned based on a statistical outlier detection method. For example, by calculating the mean and standard deviation of data such as speed, acceleration, and steering wheel angle.

[0094] Exemplarily, taking speed as an example, for speed data, if |v(t) - μ v | > kσ v , then this speed data point is considered an outlier.

[0095] Wherein, v(t) is the speed, σ v is the standard deviation of the speed, μ v is the mean of the speed, and k is a threshold set according to the actual situation (for example, k = 3).

[0096] It can be understood that in the case where the sensor data is detected as abnormal, the detected outliers can be corrected according to the continuity and rationality of the data. For example, using the average value of adjacent data points for replacement, estimating and correcting according to the vehicle dynamics model, etc. The correction method of this abnormal data can be selected according to the actual situation.

[0097] The normalization processing here refers to mapping the cleaned data such as speed, acceleration, and steering wheel angle to the interval [0, 1].

[0098] In one embodiment, the normalization formula for speed data can be:

[0099]

[0100] Wherein, v min is the minimum value of the speed data, v max is the maximum value of the speed data, and v(t) is the speed at time t.

[0101] The above-mentioned image data can normalize the image pixels to a specific mean and standard deviation through standardization processing to improve the efficiency of the subsequent deep learning model in processing image data.

[0102] Assuming the image pixel value is x ij (i, j are pixel positions), the mean μ image and standard deviation σ image, and then perform a normalization transformation.

[0103] Among them, the formula for the normalization transformation can be:

[0104]

[0105] It should be understood that in data fusion and synchronization processing, for different types of data collected at the same moment, synchronization processing is performed according to timestamps to ensure the temporal consistency of the data.

[0106] The weighted average method can be used to fuse the data of different sensors. Let the weight of the speed sensor data be w1, and the weight of the speed estimated by the object detection algorithm in the image data be w2 (w1 + w2 = 1). Then the fused speed data can be: v fused (t) = w1v(t) + w2v image (t).

[0107] Among them, v image (t) is the speed estimated from the image data.

[0108] In the above implementation process, before extracting vehicle behavior features, preprocessing methods such as cleaning, correcting, and normalizing vehicle driving data can be performed first, which can reduce abnormal data in vehicle driving data, improve the accuracy of vehicle driving data, and at the same time, can also reduce the processing difficulty of vehicle driving data.

[0109] In a possible implementation manner, step 203 includes: classifying vehicle behavior features through a fully connected neural network structure; determining the common features of vehicle behavior through a fully connected neural network structure, and determining the anomaly score through a fully connected layer.

[0110] Among them, the activation function of the hidden layer of the fully connected neural network structure is the Relu function, the output layer is the softmax function, and the loss function is the cross-entropy loss function.

[0111] Exemplarily, assume that the number of neurons in the input layer of the fully connected neural network structure is equal to the dimension d of the vehicle behavior feature vector Γ, there are l hidden layers, the number of neurons in the kth hidden layer is nk (k = 1, 2,..., l), and the number of neurons in the output layer is equal to the number of vehicle behavior categories C.

[0112] The activation function of the hidden layer here can be expressed as:

[0113] σ(x) = max(0, x);

[0114] The softmax function can be expressed as:

[0115]

[0116] Among them, z i is the input of the i-th neuron in the output layer, and is the predicted probability that the vehicle behavior belongs to the i-th category.

[0117] Furthermore, the forward propagation formula of the fully connected neural network structure can be:

[0118] h0 = Γ;

[0119] h k = σ(W k h k-1 + b k )(k = 1, 2, …, l);

[0120] y class = softmax(W l-1 h l + b l+1 );

[0121] Among them, W k is the weight matrix of the k-th hidden layer, b k is the bias vector of the k-th hidden layer, h0 is the input layer, Γ is the input data, h k is the hidden layer, h k-1 is the activation output of the (k - 1)-th layer, y class is the output layer, W l-1 is the weight matrix of the output layer, h l is the activation output of the last hidden layer, b l+1 is the bias vector of the output layer.

[0122] The cross-entropy loss function here can be expressed as:

[0123]

[0124] Among them, N is the number of samples, is the true behavior category label of the i-th sample, is the encoding vector of the i-th sample.

[0125] The loss function of the above anomaly monitoring is the mean square error loss function. y

[0126] In one embodiment, when performing anomaly monitoring, the anomaly detection part shares some network layers with the feature classification part to learn the common features of vehicle behaviors and predicts the anomaly scores through other fully connected layers.

[0127] Assume that the output of the shared network layer is h shared , the weight matrix of the anomaly score prediction layer is W ano , and the bias vector is b ano, the abnormal behavior score prediction formula can be:

[0128] y anomaly = W ano h shared + b ano ;

[0129] The loss function for abnormal behavior detection adopts the mean squared error loss function and can be expressed as:

[0130]

[0131] where is the true abnormal score of the i-th sample. If the sample is normal behavior, If it is abnormal behavior, a corresponding positive value is assigned according to the severity of the abnormality.

[0132] It should be understood that in the case of joint training and optimization, in order to optimize the vehicle behavior classification and abnormal behavior detection tasks simultaneously, the loss functions of the two tasks can be weighted and summed to obtain the total loss function:

[0133] L = λL class + (1 - λ)L anomaly ;

[0134] where λ is the weight coefficient for weighing the importance of the two tasks (which can be adjusted according to actual needs. For example, λ = 0.7 indicates that the behavior classification task is relatively more important).

[0135] It can be understood that after classifying and abnormally monitoring the vehicle behavior characteristics, the model parameters of the multi-task deep learning model can also be updated. During the training process, continuously adjusting the model parameters can improve the accuracy of vehicle behavior classification and reduce the error of abnormal behavior detection at the same time.

[0136] Optionally, the total loss function can be optimized by the gradient descent algorithm, and then the model parameters are updated. For example, optimization algorithms such as stochastic gradient descent and Adam. The method for optimizing the total loss function can be selected according to the actual situation.

[0137] The above task weight coefficient can be dynamically adjusted according to the vehicle driving scenario (such as highway, urban road, etc.) and / or traffic conditions (such as congestion, smoothness, etc.). For example, in a traffic congestion scenario, the importance of the abnormal detection task may increase, and the value of λ is appropriately reduced to make the model pay more attention to abnormal behavior detection; on a smooth road, the weight of the behavior classification task can be relatively increased to ensure accurate identification of vehicle behavior types. Through the above dynamic adjustment method, the multi-task deep learning model can better adapt to different vehicle driving situations and improve the accuracy and reliability of vehicle behavior analysis and abnormal detection.

[0138] In the above implementation process, by adopting a multi-task deep learning model to classify vehicle behavior characteristics and determine anomaly scores, the characteristic differences and correlations of vehicles under different behavior modes can be learned, thereby improving the accuracy and reliability of vehicle behavior analysis and anomaly detection.

[0139] In a possible implementation, after step 203, the method further includes: determining a warning level according to the classification category and / or anomaly score corresponding to the vehicle behavior characteristics; and determining a warning push range and method according to the warning level and the current location of the vehicle.

[0140] The warning level here can be determined by the warning decision sub-model. Among them, the input of the vehicle behavior analysis structure is the vehicle behavior analysis result, and the output is the warning level (for example, no warning, low-level warning, medium-level warning, and high-level warning, etc.).

[0141] In one embodiment, the warning level may be determined based on the vehicle behavior abnormality score and a preset threshold.

[0142] For example, when y anomaly <T1时,车辆行为被认为正常,L=0;当T1≤y anomaly <T2时,发出低级别预警,L=1;当y anomaly When T2 is greater than or equal to 2, a medium-level warning (such as abnormal behavior that persists for a certain period of time or occurs simultaneously with other abnormal behaviors) or a high-level warning (such as behavior that seriously affects traffic safety, such as driving against traffic at a high speed) is issued, and L = 2.

[0143] Among them, y anomaly is the abnormality score, T1 is the first preset threshold, T2 is the second preset threshold, L=0 means no warning, L=1 means low warning, and L=2 means high warning.

[0144] It is understandable that if the warning level is determined according to the classification category, different initial warning level weights can be set according to different vehicle behavior categories.

[0145] For example, for speeding behavior, if the speed exceeds a certain percentage of the speed limit (such as 20%), the initial warning level weight can be set as: w speeding =0.5; for retrograde behavior, the initial warning level weight can be set to: w reversing =0.8.

[0146] Further adjustments can be made based on the abnormality score, and the calculation formula for the final warning level L can be:

[0147]

[0148] Among them, w classThe initial warning level weight determined according to the classification category.

[0149] The above-mentioned determination of the warning push range and method based on the warning level and the current position of the vehicle can be achieved through the following methods:

[0150] Assume that the position coordinates of the vehicle are (x, y), and the warning area radius is r (which can be dynamically adjusted according to the warning level. For example, for low-level warnings, r = 100 meters, for medium-level warnings, r = 500 meters, for high-level warnings, r = 1000 meters, etc.). For other vehicles (x i , y i ) within the warning area, calculate the distance between it and the abnormal vehicle If d ≤ r, then push a warning message to this vehicle.

[0151] Optionally, the warning push method can include voice prompts (such as through the in-vehicle voice system), text displays (such as on the in-vehicle display screen), video displays (such as on the in-vehicle display screen), and sending notifications to the traffic management department (such as through a dedicated communication network), etc. The warning push method can be selected according to the actual situation.

[0152] For different warning levels here, the corresponding content and urgency of the pushed warnings are different. For example, the content of a high-level warning message is "Emergency! The vehicle [license plate number] ahead is performing [abnormal behavior], please drive carefully!", and the content of a low-level warning message can be "Attention: The vehicle [license plate number] ahead may have [abnormal behavior], please pay attention."

[0153] In one embodiment, a feedback mechanism can be set in the warning decision sub-model to collect feedback from drivers and the traffic management department on the warning information. Furthermore, the warning decision sub-model can be updated according to the feedback information.

[0154] For example, the idea of reinforcement learning, such as the Q-learning algorithm, can be adopted. Regarding the warning decision as an action and the warning effect (such as whether an accident is avoided) as a reward. Adjust the parameters of the warning decision model according to the reward, so that the warning decision sub-model can continuously optimize the warning strategy, thereby improving the accuracy and effectiveness of the warning.

[0155] Assume that the Q-function is Q((y anomaly, y class )), L), which is used to represent the expected long-term reward of taking the warning level L under the vehicle behavior state (y anomaly, y class ).

[0156] Among them, the value of the Q-function can be continuously updated to select the corresponding warning level. The update formula can be:

[0157]

[0158] Among them, α is the learning rate, γ is the discount factor, and r is the reward obtained according to the feedback.

[0159] It should be understood that by updating the model through the above feedback mechanism, the safety warning and information push system can continuously adapt to the actual situation and improve the accuracy of vehicle warning push.

[0160] For the convenience of understanding, the following describes the specific implementation process in the embodiments of the present application through an embodiment:

[0161] Such as Figure 3 shown, the embodiments of the present application may include the following steps;

[0162] Step 1, vehicle driving data collection and preprocessing: Specifically: collect vehicle driving data, clean the vehicle driving data and perform outlier processing, perform normalization and standardization processing on the cleaned and outlier-processed vehicle driving data, and finally perform data fusion and synchronization processing.

[0163] Step 2, vehicle behavior feature extraction: Specifically: construct a feature extraction framework based on multi-modal data, then extract sensor data features and image data features, and finally fuse multi-modal features.

[0164] Step 3, vehicle behavior analysis and anomaly detection: Specifically: construct a multi-task learning model, perform vehicle behavior classification and abnormal behavior detection based on the multi-task learning model and multi-modal features, and finally perform joint training and optimization.

[0165] Step 4, safety warning and information push: Specifically: construct a warning decision model, determine the warning level based on the warning decision model, determine the information push strategy according to the warning level, and perform information push based on the corresponding information push strategy.

[0166] In the above implementation process, by determining the warning level according to the classification category and / or anomaly score, and determining the warning push range and method according to the warning level and the current position of the vehicle, the limitations of the existing vehicle behavior detection and warning methods can be solved, and the accuracy, scientificity and flexibility of the warning can be improved.

[0167] Based on the same inventive concept, the embodiments of the present application also provide a vehicle behavior monitoring device corresponding to the vehicle behavior monitoring method. Since the principle of solving problems by the device in the embodiments of the present application is similar to that of the foregoing vehicle behavior monitoring method embodiments, the implementation of the device in this embodiment can refer to the description in the method embodiments above, and the repeated parts will not be elaborated.

[0168] Please refer to Figure 4, which is a schematic diagram of the functional modules of the vehicle behavior monitoring device provided by an embodiment of the present application. Each module in the vehicle behavior monitoring device in this embodiment is used to execute each step in the above method embodiment. The vehicle behavior monitoring device includes a preprocessing module 301, an extraction module 302, and a monitoring module 303; wherein,

[0169] The preprocessing module 301 is used to preprocess vehicle driving data; wherein, the vehicle driving data includes sensor data and image data.

[0170] The extraction module 302 is used to extract vehicle behavior features from the vehicle driving data; wherein, the vehicle behavior features are configured to fuse multi-modal information features.

[0171] The monitoring module 303 is used to classify and perform anomaly monitoring on the vehicle behavior features through a multi-task deep learning model.

[0172] In a possible implementation manner, the extraction module 302 is further used to: extract sensor data features from the sensor data through a recurrent neural network structure; extract image data features from the image data through a convolutional neural network; and perform feature fusion on the sensor data features and the image data features by using a weighted summation method to obtain the vehicle behavior features.

[0173] In a possible implementation manner, the extraction module 302 is specifically used to: input the sensor data at each moment into the recurrent neural network structure and calculate the hidden state at each moment; wherein, a gating mechanism is included in the recurrent neural network structure; and use the hidden state at the final moment as the sensor data features of the sensor data.

[0174] In a possible implementation manner, the extraction module 302 is specifically used to: perform a convolution operation on the image data and a convolution kernel in the convolutional neural network to extract a feature map of the image data; multiply the attention weight by the feature map to obtain a weighted feature map; and process the weighted feature map through a pooling layer and a fully connected layer to obtain the image data features.

[0175] In a possible implementation manner, the preprocessing module 301 is further used to: calculate the mean and standard deviation of the sensor data, and determine whether the sensor data is abnormal according to the mean and the standard deviation; in the case where the sensor data is abnormal, correct the abnormal sensor data; perform normalization processing on the corrected sensor data; normalize the image data to a specific mean and standard deviation; and perform data fusion on the normalized sensor data and the normalized image data.

[0176] In a possible implementation, the monitoring module 303 is further configured to: classify the vehicle behavior features through a fully connected neural network structure; wherein, the activation function of the hidden layer of the fully connected neural network structure is the Relu function, the output layer is the softmax function, and the loss function is the cross-entropy loss function; determine the common features of vehicle behavior through the fully connected neural network structure, and determine the anomaly score through the fully connected layer; wherein, the loss function of anomaly monitoring is the mean squared error loss function.

[0177] In a possible implementation, the vehicle behavior monitoring device further includes an early warning module, configured to determine the early warning level according to the classification category corresponding to the vehicle behavior feature and / or the anomaly score; determine the early warning push range and method according to the early warning level and the current position of the vehicle.

[0178] In addition, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, it executes the steps of the vehicle behavior monitoring method described in the above method embodiment.

[0179] The computer program product of the vehicle behavior monitoring method provided by the embodiment of the present application includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the vehicle behavior monitoring method described in the above method embodiment. For details, reference can be made to the above method embodiment, which will not be elaborated here.

[0180] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0181] In addition, in each embodiment of the present application, each functional module may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

[0182] If the above-mentioned function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes. It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, the elements defined by the statement "including..." do not exclude the existence of additional identical elements in the process, method, article or device including the said elements.

[0183] The foregoing is only the preferred embodiment of the present application and is not used to limit the present application. For those skilled in the art, the present application may have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0184] The foregoing is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present application, and all of them should be covered by the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A vehicle behavior monitoring method, characterized in that, Including: Preprocessing vehicle driving data; wherein, the vehicle driving data includes sensor data and image data; Extracting vehicle behavior features from the vehicle driving data; wherein, the vehicle behavior features are configured to fuse multi-modal information features; Classifying and abnormally monitoring the vehicle behavior features through a multi-task deep learning model.

2. The method according to claim 1, wherein The extracting vehicle behavior features from the vehicle driving data includes: Extracting sensor data features from the sensor data through a recurrent neural network structure; Extracting image data features from the image data through a convolutional neural network; Performing feature fusion on the sensor data features and the image data features by using a weighted summation method to obtain the vehicle behavior features.

3. The method according to claim 2, wherein The extracting sensor data features from the sensor data through a recurrent neural network structure includes: Inputting the sensor data at each moment into the recurrent neural network structure and calculating the hidden state at each moment; wherein, a gating mechanism is included in the recurrent neural network structure; Taking the hidden state at the final moment as the sensor data features of the sensor data.

4. The method according to claim 2, wherein The extracting image data features from the image data through a convolutional neural network includes: Performing a convolution operation on the image data and a convolution kernel in the convolutional neural network to extract a feature map of the image data; Multiplying the attention weight by the feature map to obtain a weighted feature map; Processing the weighted feature map through a pooling layer and a fully connected layer to obtain the image data features.

5. The method according to claim 1, wherein The preprocessing vehicle driving data includes: Calculating the mean and standard deviation of the sensor data and determining whether the sensor data is abnormal according to the mean and the standard deviation; Correcting the abnormal sensor data in the case where the sensor data is abnormal; Performing normalization processing on the corrected sensor data; Normalizing the image data to a specific mean and standard deviation; Performing data fusion on the normalized sensor data and the normalized image data.

6. The method according to claim 1, characterized in that, The classifying and abnormally monitoring the vehicle behavior features through a multi-task deep learning model includes: Classifying the vehicle behavior features through a fully connected neural network structure; wherein, the activation function of the hidden layer of the fully connected neural network structure is the Relu function, the output layer is the softmax function, and the loss function is the cross-entropy loss function; Determining the common features of vehicle behaviors through the fully connected neural network structure and determining an abnormal score through a fully connected layer; wherein, the loss function for abnormal monitoring is the mean squared error loss function.

7. The method according to claim 1, characterized in that, After the classifying and abnormally monitoring the vehicle behavior features through a multi-task deep learning model, the method further includes: Determining a warning level according to the classification category and / or abnormal score corresponding to the vehicle behavior features; Determining a warning push range and method according to the warning level and the current position of the vehicle.

8. A vehicle behavior monitoring device, characterized in that, Including: A preprocessing module for preprocessing vehicle driving data; wherein, the vehicle driving data includes sensor data and image data; An extraction module for extracting vehicle behavior features from the vehicle driving data; wherein the vehicle behavior features are configured to fuse multi-modal information features; A monitoring module for classifying and abnormally monitoring the vehicle behavior features through a multi-task deep learning model.

9. An electronic device, characterized in that, Comprising: A processor and a memory, the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the machine-readable instructions are executed by the processor to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is run by the processor, it performs the steps of the method according to any one of claims 1 to 7.

Images