Automatic driving anomaly detection method based on video frame prediction
By employing a lightweight network architecture and data preprocessing, combined with PSNR rule scores and optical flow field characteristics, the limitations of computing power and adaptability to complex scenarios on vehicle platforms are addressed, enabling efficient anomaly detection for autonomous driving and improving detection accuracy and real-time performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-03-10
AI Technical Summary
Existing frame prediction models have complex architectures, large numbers of parameters, and high computational overhead, making them unsuitable for the limited computing power of in-vehicle platforms and resulting in insufficient real-time performance. Existing anomaly detection algorithms have poor generalization ability in autonomous driving and cannot cover complex scenarios such as extreme weather and sudden behavior. The connection logic between frame prediction and anomaly judgment is simple, resulting in a low anomaly recognition rate.
A lightweight network architecture is adopted, combining Gaussian filtering and Hessian matrix for data preprocessing, and a lightweight frame prediction network model is used for video frame prediction. Anomaly detection is performed using PSNR rule scores and optical flow field features, and the anomaly detection threshold is dynamically adjusted to adapt to different scenarios.
It achieves a lightweight architecture adapted to vehicle computing power, improves detection accuracy and real-time performance, enhances the accuracy and generalization ability of anomaly detection, adapts to complex working conditions, and meets the real-time response requirements of autonomous driving systems.
Smart Images
Figure CN121640348A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of automatic driving and computer vision, in particular to an automatic driving anomaly detection method based on video frame prediction, which is especially suitable for real-time environment perception scenarios in automatic driving systems. The method can perform frame prediction and anomaly determination on video data collected by vehicle-mounted cameras through a lightweight deep learning model, solving the contradiction between detection accuracy and real-time performance caused by video blurring and limited vehicle-mounted computing power at high speeds. BACKGROUND
[0002] With the development of automatic driving technology to high level (L3 and above), pure visual perception scheme has become the mainstream due to its low cost and flexible deployment, and its core relies on computer vision and deep learning algorithms for real-time analysis of traffic scenes. Among them, the anomaly detection technology based on video frame prediction can identify anomalies (such as pedestrians crossing, vehicles suddenly braking, and collisions) in the driving process in advance by predicting the difference between future frames and real frames, providing safety response time for the automatic driving system, and is one of the key technologies to improve driving safety. In the prior art, frame prediction has been applied in the field of monitoring video anomaly detection (such as patent CN115170997A), but its adaptability in the automatic driving scene has significant defects: first, the vehicle-mounted platform has limited computing power (compared to the server side), and existing frame prediction models mostly use complex recursive or transformation architectures (such as LSTM, Transformer), which have large parameter quantities and high computational overhead, making it difficult to meet the real-time requirements of automatic driving ≥30FPS; second, at high speeds, the video frames collected by the vehicle-mounted camera are prone to motion blur and ghosting (due to mismatch between shutter speed and vehicle motion speed), and existing technologies do not perform data preprocessing for this problem, resulting in distorted predicted frames and directly affecting the accuracy of anomaly detection; third, related research (such as Ru Haodong's “Video anomaly detection technology for driving field”) only focuses on the detection algorithm itself and does not consider the actual working conditions of automatic driving with high dynamics and strong real-time, resulting in poor model generalization ability and difficulty in adapting to complex scenes such as rain, fog, and sudden changes in light. Therefore, to address the technical pain points of limited computing power, video blurring, insufficient real-time performance, and poor generalization ability in driving scenarios, we propose an automatic driving anomaly detection method based on video frame prediction, which balances detection accuracy and real-time performance through lightweight network architecture design and targeted data preprocessing. In the experiment conducted on urban secondary arterial roads with a speed of 20-40km / h, limited field of view, and buildings and trees blocking the view, the anomaly detection rate reached more than 95%. SUMMARY
[0003] To solve the problems of the prior art, the application provides an automatic driving anomaly detection method based on video frame prediction, which solves the following technical problems: 1. The existing frame prediction model has a complex architecture (such as containing a recursive layer and a large-core convolution), a large number of parameters, and high calculation overhead, and cannot adapt to the limited computing power of a vehicle-mounted platform, resulting in insufficient real-time performance (inference speed < 20 FPS); 2. The existing anomaly detection algorithm relies on abnormal sample labeling, has poor generalization ability for long-tail anomalies (low-frequency high-risk events) in automatic driving, and cannot cover complex scenes such as extreme weather and sudden behaviors; 3. The connection logic of frame prediction and anomaly determination is simple, and the consistency of the spatiotemporal features is not combined, resulting in low anomaly recognition rate.
[0004] In order to achieve the above purpose, the application specifically adopts the following technical solutions: An automatic driving anomaly detection method based on video frame prediction comprises the following steps: step 1: video frame preprocessing - the continuous video frame sequence collected by the vehicle-mounted camera is jointly processed by Gaussian filtering and a Hessian matrix, wherein the Gaussian filtering (standard deviation = 1.2) is used to smooth the noise in the frame, and the Hessian matrix (window size 3x3) is used to enhance the edge features of the frame and suppress motion blur, to obtain a denoised and deblurred video frame image; step 2: lightweight frame prediction - the preprocessed video frame is input into a pre-trained lightweight frame prediction network model to output a predicted frame image and a peak signal-to-noise ratio (PSNR); the lightweight frame prediction network model is composed of an encoder, a translator and a decoder connected in sequence, and integrates a DynamicTanh (DyT) layer; step 3: anomaly score calculation - the PSNR value is normalized to the interval [0, 1], and the regular score of each frame is calculated by the formula = 1.2) is used to smooth the noise in the frame, and the Hessian matrix (window size 3x3) is used to enhance the edge features of the frame and suppress motion blur, to obtain a denoised and deblurred video frame image; step 2: lightweight frame prediction - the preprocessed video frame is input into a pre-trained lightweight frame prediction network model to output a predicted frame image and a peak signal-to-noise ratio (PSNR); the lightweight frame prediction network model is composed of an encoder, a translator and a decoder connected in sequence, and integrates a DynamicTanh (DyT) layer; step 3: anomaly score calculation - the PSNR value is normalized to the interval [0, 1], and the regular score of each frame is calculated by the formula
[0005] Further, the encoder in step 2 is composed of 3 layers of lightweight convolutional layers, each with a kernel size of 3x3, a stride of 1x1, and padding=1, and a DyT layer connected in series after each layer. The role of the encoder is to encode high-dimensional video frames into a low-dimensional latent space and extract intra-frame spatial features such as edges and textures. The DyT layer achieves a normalization effect similar to the formula f(x)=γ×tanh(αx)+β, without the need to calculate activation statistics, reducing memory usage and inference delay. Further, the translator in step 2 integrates a space-time attention module (STU). The STU module replaces traditional large kernel convolution with a combination of "deep convolution (3x3) + deep dilated convolution (dilation rate r=2) + 1x1 convolution". Deep convolution captures single-channel local receptive fields, deep dilated convolution connects long-range spatio-temporal dependencies, and 1x1 convolution enables feature interaction between channels. This approach reduces computational complexity by 60% while maintaining a receptive field size equivalent to a 7x7 large kernel. Further, the decoder in step 2 is composed of 3 layers of deconvolutional layers, each with a kernel size of 3x3, a stride of 2x2, and padding=1, for restoring low-dimensional latent features to high-dimensional predicted frames. A DyT layer is connected in series after each deconvolutional layer to ensure feature restoration accuracy. Further, the training process of the lightweight frame prediction network model in step 2 is as follows: using video frame sequences in normal driving scenarios as training data, adopting mean square error (MSE) as the loss function, and iterating training through the Adam optimizer (learning rate initially set to 1e-4) until the validation set PSNR stabilizes at ≥35dB, stopping training and saving the model. Further, the abnormal type identification in step 3 is achieved through spatio-temporal feature consistency determination: using the Flownet2.0 network to estimate the optical flow field of "pre-processed continuous N-frame input sequence" and "real N+1 frame", respectively, to obtain the inter-frame optical flow field of the input sequence and the reference optical flow field of the real future frame. The dimension of the optical flow field is consistent with the size of the video frame. Calculate the optical flow field difference between the predicted future frame and the real future frame, and define the optical flow loss function. Incorporate the optical flow loss into the total loss function of the model to constrain the lightweight encoding and decoding network, ensuring that the motion trajectory, speed change, and other motion patterns of the predicted frame conform to the real physical laws, and avoiding unreasonable motion prediction results such as "vehicle passing through walls" and "instantaneous movement of pedestrians".
[0006] Further, the abnormality determination threshold in step 4 supports dynamic adjustment: through a sliding window (window size 100 frames) to count the rule score distribution under normal scenes, if the average of the rule score of 5 consecutive frames <0.3, the threshold is lowered to 0.5 (adapt to low dynamic scenes); if the rule score fluctuation of 3 consecutive frames >0.2, the threshold is raised to 0.7 (adapt to high dynamic scenes), avoiding misjudgment caused by environmental interference.
[0007] Compared with the prior art, the present application provides an automatic driving anomaly detection method based on video frame prediction, which has the following beneficial effects: Lightweight architecture adapts to vehicle computing power - by replacing traditional large kernel convolution with "deep convolution + dilated convolution" and replacing normalization layer with DyT layer, the network parameter quantity is reduced to 40% of the existing model, and the inference speed is increased to >=35FPS, meeting the real-time requirements of vehicle platform (>=30FPS) and can be stably run on edge devices such as NVIDIA Jetson AGXXavier; Preprocessing improves prediction accuracy - joint preprocessing of Gaussian filtering and Hessian matrix reduces the blurriness of video frames by 45% at high speed, the prediction frame PSNR is increased to >=38dB, the anomaly detection accuracy is increased to >=96%, and the false detection rate is <3%; No abnormal sample training adapts to long-tail scenarios - only normal driving data is used to train the model, without labeling abnormal samples, the coverage of long-tail anomalies such as extreme weather (heavy rain, heavy fog), sudden behavior (pedestrian crossing, foreign object falling) reaches 92%, and the generalization ability is significantly better than the prior art; Accurate abnormal type identification - combining PSNR rule score and optical flow field spatiotemporal features can accurately distinguish three types of core anomalies: "traffic participant anomaly", "environmental anomaly" and "sensor failure", with an identification accuracy of >=94%, providing targeted response basis for automatic driving system (such as "pedestrian crossing" triggering emergency braking, "lighting mutation" only adjusting perception parameters); Dynamic threshold adapts to complex working conditions - through sliding window dynamic adjustment of abnormality determination threshold, reducing misjudgment in low dynamic scenes (such as high-speed uniform driving) and improving sensitivity in high dynamic scenes (such as urban intersections), scene adaptability is improved by 30%. BRIEF DESCRIPTION OF DRAWINGS
[0008] Figure 1 The overall flowchart of the automatic driving anomaly detection method based on video frame prediction of the present application; Figure 2 The overall structure diagram of the lightweight frame prediction network model; Figure 3 The structure diagram of the spatiotemporal attention module (STU) in the translator; Figure 4A comparison diagram of the difference between a predicted frame and a real frame Figure 5 A PSNR curve diagram of each predicted image. DETAILED DESCRIPTION
[0009] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.
[0010] EMBODIMENT As Figure 1 anomaly detection flowchart), Figure 2 a lightweight frame prediction network model overall structure), and Figure 3 an STU module), an automatic driving anomaly detection method based on video frame prediction proposed by an embodiment of the present application includes two parts of hardware environment building and software process execution, which are as follows. I. Hardware environment building Video acquisition device: a vehicle-mounted monocular wide-angle camera (model: Mobileye EyeQ6L matching camera) is selected, the resolution is set to 1920x1080, the frame rate is 30FPS, the shutter speed is 1 / 1000s (adapted to high-speed driving scene, preliminary reduction of motion blur), and the camera is installed in the center of the vehicle front windshield (with a field of view covering 120° in front), supporting real-time output of RGB format video stream; Computing platform: a vehicle-mounted edge computing unit (model: NVIDIA Jetson AGXXavier) is used, with a computing power of 32TOPS (INT8), a memory of 32GB LPDDR4, and a storage of 1TB SSD, meeting the real-time inference requirements of lightweight models; Software environment: based on Ubuntu 20.04 operating system, Python 3.8, TensorFlow 2.10 (model training and inference), OpenCV 4.6 (video frame preprocessing), PyTorch 1.12 (optical flow field calculation) are deployed, and CUDA 11.7 is configured to accelerate GPU operation.
[0011] II. Software process execution (corresponding to Figure 1 ) Step 1: Video frame preprocessing (in Figure 1 the "preprocessing module") Read a continuous video frame sequence from the vehicle's camera (take 10 consecutive frames as a group, with a frame interval of 1 / 30s), and perform Gaussian filtering using OpenCV's GaussianBlur function: set the filter kernel size to 5×5, and the standard deviation to... =1.2, smoothing high-frequency noise within the frame (such as road surface reflection and dust interference). Call a custom Hessian matrix calculation function, set the window size to 3×3, calculate the pixel grayscale changes within the frame through the second-order partial derivative, enhance edge features (such as lane lines and vehicle outlines), and suppress motion blur (such as background blur trailing when driving at high speed), and output the denoised and deblurred video frame (the size is kept at 1920×1080). Step 2: Lightweight frame prediction ( Figure 1 "Frame Prediction Module", corresponding to Figure 2 ) Model Loading: Load the pre-trained lightweight frame prediction network model (file format: .h5), the model structure is as follows Figure 2 As shown, it includes an encoder (3 convolutional layers), a translator (including an STU module), and a decoder (3 deconvolutional layers), with each convolutional / deconvolutional layer followed by a DyT layer. Encoder parameters: Convolutional layer 1 (3×3, stride 1×1, padding=1, output channels 64) → DyT layer (α=1.0, β=0.0, γ=0.5, initial value) → Convolutional layer 2 (3×3, stride 1×1, padding=1, output channels 128) → DyT layer → Convolutional layer 3 (3×3, stride 1×1, padding=1, output channels 256) → DyT layer; Translator STU module (corresponding) Figure 3 ): Input 256-channel feature map → depthwise convolution (3×3, single-channel convolution, no channel interaction) → depthwise dilated convolution (3×3, dilation rate r=2, covering remote spatiotemporal features) → 1×1 convolution (256 channels, achieving channel interaction) → output 256-channel feature map; Decoder parameters: Deconvolution layer 1 (3×3, stride 2×2, padding=1, output channels 128) → DyT layer → Deconvolution layer 2 (3×3, stride 2×2, padding=1, output channels 64) → DyT layer → Deconvolution layer 3 (3×3, stride 2×2, padding=1, output channels 3) → DyT layer, finally outputting 1920×1080 3-channel prediction frames; Frame prediction inference: The preprocessed 10-frame sequence is input into the model, and the model outputs the predicted 11th frame. Simultaneously, the peak signal-to-noise ratio (PSNR) of the predicted frame and the subsequently acquired true 11th frame is calculated using the following formula: In the normal scenario, the PSNR is stable at 32-35dB in this embodiment; Model training supplement: the training data uses normal driving videos (including high-speed, urban, and suburban scenes, a total of 500 hours, about 5.4 million frames) in public automatic driving datasets (KITTI, WaymoOpenDataset), without abnormal sample labeling; during training, 10 consecutive frames are used as input and the 11th frame is used as the true label, using the MSE loss function, Adam optimizer (initial learning rate 1e-4, decaying to 0.8 of the previous round every 10 rounds), and verifying the PSNR≥35dB after 100 iterations, stopping training and saving the model.
[0012] Step 3: Use the Flownet2.0 network to estimate the optical flow field of the "pre-processed continuous N-frame input sequence" and the "true N+1 frame", respectively, to obtain the inter-frame optical flow field F_in of the input sequence and the reference optical flow field F_ref of the true future frame. The dimension of the optical flow field is consistent with the size of the video frame, and each pixel point corresponds to a two-dimensional optical flow vector (u, v), which represents the horizontal and vertical motion speed, respectively; calculate the optical flow field difference between the predicted future frame and the true future frame, and define the optical flow loss function as L_flow=||F_pred-F_ref||2, where F_pred is the optical flow field corresponding to the predicted future frame, which is obtained by inputting the predicted frame and the last frame of the input sequence into Flownet2.0; integrate the optical flow loss L_flow into the total loss function of the model, and the total loss function is updated as L_total=L_mse+λ×L_flow (λ is the optical flow loss weight, the value range is 0.3-0.7, and the optimal value is 0.5), and the generator (i.e. lightweight coding and decoding network) is constrained by the optical flow loss, to ensure that the motion trajectory, speed change and other motion patterns of the predicted frame meet the true physical law, and avoid unreasonable motion prediction results such as "vehicle wall penetration" and "instantaneous movement of pedestrians". Through multi-loss joint optimization, the predicted frame is highly close to the true frame in visual intensity, edge details, and motion consistency, and the PSNR is improved to ≥35dB.
[0013] Step 4: Abnormality judgment and output Figure 1 Abnormality judgment module Input the predicted frame and the true frame into the discriminator, and normalize the PSNR value of all videos to a specific range [0, 1], and calculate the score using the following formula: The score exceeding the threshold value indicates an anomaly, and the type of anomaly is determined in combination with the optical flow field information. The initial threshold value is set to 0.6 based on 50 hours of normal high-speed driving data statistics (i.e., s > 0.6 is determined as an anomaly); the dynamic threshold value is adjusted: the sliding window (window size 100 frames) is used to statistically analyze the score distribution: if the average s of 5 consecutive frames is <0.2 (low dynamic scene, such as high-speed uniform driving), the threshold value is lowered to 0.5 to reduce false positives caused by slight environmental interference (such as short shadows); if the s fluctuation of 3 consecutive frames is >0.2, the threshold value is raised to 0.7 to improve anomaly sensitivity; the type of anomaly is identified in combination with the optical flow field difference: if there is a local mutation in the optical flow field (the deviation of the optical flow vector in a certain area from the overall motion direction is >90°), and s >0.6, it is determined as “sudden behavior of traffic participants” (such as pedestrians crossing, vehicles turning sharply), and the confidence level (based on s value, s=0.8, confidence level 90%) is output; if the overall optical flow field is blurred and s >0.6, it is determined as “environmental anomaly” (such as rain, fog, camera obstruction), and the suggestion “reduce speed and turn on fog lights” is output. The result output: the abnormal result (type, response suggestion) is transmitted to the automatic driving control system through the CAN bus, triggering the corresponding action (such as “pedestrian crossing” triggering emergency braking, braking response delay <100ms).
[0014] As shown in Figure 2 , in some embodiments, the encoder of the lightweight frame prediction network can be replaced by a 4-layer convolution (add 1 layer of 3x3 convolution, output channel 512), at this time the model feature extraction capability is improved by 15%, but the inference speed may decrease, which is suitable for urban scenes with higher accuracy requirements.
[0015] As shown in Figure 3 , in some embodiments, the depth expansion convolution expansion rate of the STU module can be adjusted to 3 (equivalent to 9x9 large kernel receptive field), at this time the remote spatiotemporal feature capture capability is improved by 20%, but the calculation amount increases by 10%, which can be adapted to high-speed scenes with large vehicle spacing (which requires to capture abnormal behaviors of vehicles at a long distance).
[0016] As shown in Figure 1 , in some embodiments, the video frame preprocessing can add a “grayscale + edge enhancement” step (extracting edges through the Canny function of OpenCV), at this time the preprocessing time increases by 5ms, but the motion blur suppression effect is improved by 30%, which is suitable for extreme foggy scenes such as heavy rain.
[0017] As shown in Figure 2 , in some embodiments, the parameters of the DyT layer can be dynamically adjusted: during training, the scene is divided into batches for optimization (high-speed scene =1.2, urban scene =0.8), at this time, the model is improved by 2-3dB in PSNR in different scenes, and the generalization ability is further enhanced.
[0018] As shown in the formula (1), in some embodiments, the abnormal score calculation can add a "structural similarity index (SSIM)" weight: according to Figure 1 As shown in the formula (1), in some embodiments, the abnormal score calculation can add a "structural similarity index (SSIM)" weight: according to Calculate the comprehensive score (SSIM is the structural similarity of the predicted frame and the real frame, the range is [0, 1]), at this time, the accuracy of the abnormal detection is slightly improved.
[0019] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for limiting the present application, although the foregoing embodiments of the present application are described in detail, for those skilled in the art, it still can be modified, or the equivalent replacement of part of the technical features recorded in the foregoing embodiments. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.
Claims
1. An automatic driving anomaly detection method based on video frame prediction, characterized in that, The method comprises the following steps: Step 1: preprocessing the continuous video frame sequence in the automatic driving scene, which comprises noise smoothing and motion smear suppression; Step 2: inputting the preprocessed video frame sequence into a pre-trained lightweight codec network to generate a future frame prediction result, wherein the lightweight codec network comprises an encoder, a spatiotemporal feature capturing module and a decoder, and is integrated with an adaptive activation layer; Step 3: estimating the optical flow field of the input frame sequence and the real future frame by using Flownet, and ensuring that the motion mode of the predicted future frame conforms to the real physical law by using the optical flow loss constraint generator; by combining the intensity loss and the gradient loss, the predicted frame is closer to the real frame in terms of visual intensity, edge details and the like, so that a clearer predicted frame is obtained; Step 4: inputting the predicted frame and the real frame into a discriminator to calculate a multi-dimensional error, and determining driving abnormality by using a dynamic threshold algorithm; Step 5: combining the optical flow field feature and the comprehensive error to determine the abnormal type. 2.The video frame prediction based automatic driving anomaly detection method of claim 1, wherein, The preprocessing of step 1 comprises Gaussian filtering and Hessian matrix joint processing: the standard deviation of Gaussian filtering is 1.2 for noise smoothing; the window size of Hessian matrix is 3x3 for edge enhancement and motion smear suppression. 3.The video frame prediction based automatic driving anomaly detection method of claim 1, wherein, The multi-scale feature extraction module comprises at least three parallel feature extraction branches, the convolution kernel sizes of the branches are 3x3, 5x5 and 7x7 respectively, and the feature maps output by the branches are spliced in the channel dimension through a feature fusion layer; the channel attention mechanism is realized by using a global average pooling and a fully connected layer to dynamically adjust the weights of each channel of the spliced feature map. 4.The method of claim 1, wherein, The multi-dimensional error comprises pixel difference and motion consistency deviation, and the dynamic threshold algorithm is a self-adaptive threshold adjustment algorithm based on Gaussian distribution, which dynamically updates the abnormality determination threshold according to the normal frame error distribution of the current scene. 5.The video frame prediction based automatic driving anomaly detection method of claim 1, wherein, The abnormal type comprises sudden obstacles, environmental abnormalities and traffic behavior abnormalities, and the confidence is a value between 0 and 1, which is used to represent the reliability of the determination result. 6.The video frame prediction based automatic driving anomaly detection method of claim 1, wherein, The method does not need to label abnormal samples, but only needs to train the deep learning network by using normal driving data, the frame prediction inference speed is greater than or equal to 30FPS, the abnormality detection accuracy is greater than or equal to 95%, and the method is suitable for complex road conditions such as rain, fog and light mutation. 7.The video frame prediction based automatic driving anomaly detection method of claim 1, wherein, In step 5, if the optical flow field has local mutation and the discriminator output error is greater than 0.6, it is determined as "sudden behavior of traffic participants"; if the optical flow vector variance is less than 0.1 and the pixel-level error is greater than 0.3, it is determined as "environmental abnormality"; if the optical flow field has overall direction abnormality, it is determined as "traffic behavior abnormality"; after identifying the abnormal type, the abnormal type is output.