A video monitoring system based on image feature analysis

Abstract

The present application relates to the technical field of video monitoring, in particular to a video monitoring system based on image feature analysis.The present application comprises a multi-modal video signal acquisition module, an environmental adaptability signal enhancement module, an environmental adaptability detection and identification module, and an abnormal behavior monitoring module;first, multi-modal video signals and environmental data are acquired to generate multi-modal video frame sequences and construct environmental state vectors; based on the environmental state vectors, an optimal signal enhancement strategy subset is selected from a signal enhancement strategy set to process the multi-modal video frames to obtain optimized multi-modal video frames; then multi-modal signal features are extracted, and a fusion feature vector is generated; the fusion feature vector is subjected to target detection to extract vehicle regions and license plate regions, and vehicle types and license plate numbers are further identified; finally, vehicle tracking is performed to obtain vehicle trajectory information, abnormal behaviors are identified, and alarms and information storage are performed.The present application improves the reliability of the video monitoring system under complex weather conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video monitoring, in particular to a video monitoring system based on image feature analysis. BACKGROUND

[0002] With the acceleration of urbanization, the increase of traffic flow and the increase of the number of motor vehicles, vehicle access management is increasingly important in traffic order maintenance and public safety guarantee. Real-time monitoring of vehicles not only needs to record the basic information of vehicle access, but also needs to identify and intervene abnormal behaviors in time. However, the monitoring effectiveness and accuracy of existing monitoring systems under complex weather conditions are often limited.

[0003] Complex weather conditions are one of the main challenges faced by vehicle monitoring systems. Rain and snow weather can produce raindrops, snowflakes or haze interference in the video, and the contrast of video images is reduced under low light environment, which will affect the capturing ability of monitoring equipment and cannot provide clear pictures, affecting the identification of vehicles and the processing of subsequent data. In addition, vehicle access monitoring requires high accuracy of target identification, not only to identify vehicle features, but also to extract license plate information. However, traditional systems rely on a single signal source, and their performance under adverse weather conditions is usually not satisfactory. At the same time, these systems lack flexible signal optimization and environmental adaptation ability, and are difficult to dynamically adjust video images according to different weather conditions and vehicle characteristics. In the face of the above problems, a vehicle access video monitoring technology that can adapt to complex weather environment is needed.

[0004] To this end, a video monitoring system based on image feature analysis is proposed. SUMMARY

[0005] The present application aims to provide a video monitoring system based on image feature analysis. The present application comprises a multi-modal video signal acquisition module, an environmental adaptability signal enhancement module, an environmental adaptability detection and identification module, and an abnormal behavior monitoring module. First, multi-modal video signals and environmental data are collected to generate multi-modal video frame sequences and construct environmental state vectors. Based on the environmental state vectors, the optimal signal enhancement strategy subset is selected from the signal enhancement strategy set to process the multi-modal video frames to obtain optimized multi-modal video frames. Then, multi-modal signal features are extracted, and a fusion feature vector is generated. The fusion feature vector is used for target detection to extract vehicle and license plate regions, and further identify vehicle types and license plate numbers. Finally, the vehicle is tracked to obtain vehicle trajectory information, and abnormal behaviors are identified based on the vehicle trajectory information to alarm.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0007] A video monitoring system based on image feature analysis comprises:

[0008] The multimodal video signal acquisition module acquires multimodal video signals by integrating visible light video signals, infrared video signals, and thermal imaging video signals, while simultaneously acquiring environmental data. It timestamps the multimodal video signals to generate a multimodal video frame sequence and constructs an environmental state vector based on the environmental data.

[0009] An environmentally adaptive signal enhancement module constructs a signal enhancement strategy set, selects an optimal subset of signal enhancement strategies from the signal enhancement strategy set based on the environmental state vector, and processes the multimodal video frames using the optimal subset of signal enhancement strategies to obtain optimized multimodal video frames.

[0010] An environmental adaptability detection and recognition module includes a feature fusion unit, a target detection unit, and a target recognition unit. The feature fusion unit extracts multimodal signal features based on the optimized multimodal video frames and generates a fused feature vector by combining it with the environmental state vector. The target detection unit extracts vehicle regions and license plate regions based on the fused feature vector through a target detection network. The target recognition unit identifies the vehicle type and license plate number based on the vehicle regions and license plate regions.

[0011] The abnormal behavior monitoring module includes a monitoring and tracking unit and a behavior determination unit; the monitoring and tracking unit tracks vehicles in the multimodal video frame sequence to obtain vehicle trajectory information; the behavior determination unit identifies abnormal behavior based on the vehicle trajectory information, issues an alarm, and stores the corresponding vehicle information and abnormal behavior information.

[0012] Further, selecting the optimal subset of signal enhancement strategies from the set of signal enhancement strategies based on the environmental state vector includes:

[0013] The contrast, sharpness, and noise level are calculated based on the multimodal video frames to obtain the original quality vector;

[0014] Based on the original quality vector and the environmental state vector, a predicted enhanced quality vector is obtained after each signal enhancement strategy processes the multimodal video frame; the predicted enhanced quality vector includes predicted contrast, predicted sharpness, and predicted noise level.

[0015] Calculate the enhancement strategy score based on the predicted enhanced quality vector:

[0016]

[0017] Among them, score(p i (I, E) represents the score of the enhancement strategy; p irepresents the ith signal enhancement strategy in the signal enhancement strategy set; I represents the multimodal video frame; E represents the environment state vector; represents the p i the predicted contrast for processing I; represents the predicted definition; represents the predicted noise level; R(p i , E) represents the correlation of the signal enhancement strategy p i with the environment state vector; w1, w2, w3, and w4 represent weight parameters;

[0018] According to the enhancement strategy score, all the signal enhancement strategies are sorted from large to small, and the top m signal enhancement strategies are selected as the optimal signal enhancement strategy subset of the multimodal video frame.

[0019] Further, the environmental complexity is calculated according to the environment state vector, when the environmental complexity is greater than the maximum complexity threshold, the value of m is the number of the signal enhancement strategies in the signal enhancement strategy set; when the environmental complexity is less than the minimum complexity threshold, the value of m is 1.

[0020] Further, different weather environment data and image sample data under corresponding weather environment are collected to construct a training set; for each signal enhancement strategy, an enhancement prediction model is established by a machine learning method and trained by the training set; the original quality vector and the environment state vector are input into the enhancement prediction model to obtain the predicted enhancement quality vector of each signal enhancement strategy.

[0021] Further, the feature fusion unit comprises:

[0022] According to the optimized multimodal video frame at the same time, multimodal signal features are extracted;

[0023] According to the multimodal signal features and the environment state vector, multimodal signal feature fusion is performed to obtain the fusion feature vector; the formula of the multimodal signal feature fusion is:

[0024]

[0025] wherein, F fusion represents the fusion feature vector; M represents the number of multimodal signal features; σ() represents an activation function; W i represents a weight matrix; E represents the environment state vector; b i represents a bias term; W i，s represents a spatial attention weight matrix; W i，c represents a channel attention weight matrix; Fi represents the i-th multi-modal signal feature.

[0026] Further, the target detection unit and the target recognition unit comprise:

[0027] The target detection unit generates the vehicle region by adopting a Mask R-CNN model for target detection based on the fusion feature vector; and generates the license plate region by performing target detection again after extracting features from the vehicle region;

[0028] The target recognition unit extracts visual features of the vehicle region by adopting a deep convolutional neural network, classifies the vehicle type by a multi-layer perceptron, performs character recognition on the license plate region by a Transformer model, and obtains the license plate number.

[0029] Further, the environment adaptability detection and recognition module further comprises a region relocation unit, constructs an edge signal enhancement strategy set, selects an optimal edge signal enhancement strategy subset from the edge signal enhancement strategy set based on an environment state vector, processes the optimized multi-modal video frame through the optimal edge signal enhancement strategy subset to obtain an updated optimized multi-modal video frame, and generates a relocated vehicle region and a relocated license plate region based on the updated optimized multi-modal video frame through the feature fusion unit and the target detection unit.

[0030] Further, the target recognition unit further comprises a license plate color recognition subunit, performs color feature analysis on the license plate region, extracts color space features, calculates a green component proportion, determines that the vehicle is a new energy vehicle when the green component proportion is greater than a preset green threshold, generates a relocated vehicle region to replace the vehicle region and a relocated license plate region to replace the license plate region through the region relocation unit.

[0031] Further, the monitoring and tracking unit comprises:

[0032] The vehicle region of each video frame is subjected to information extraction to obtain a frame sequence number and a vehicle state feature; the vehicle state feature comprises a vehicle region coordinate, a vehicle region size, and speed information.

[0033] The predicted vehicle state feature in the current video frame is predicted by Kalman filtering based on the vehicle state feature of the previous video frame.

[0034] The data association result is obtained by utilizing the Hungarian algorithm to perform data association according to the vehicle state feature of the current video frame and the predicted vehicle state feature.

[0035] updating the vehicle state feature of the current video frame according to the data association result, to obtain the vehicle trajectory information; the vehicle trajectory information includes a vehicle tracking ID of each vehicle in all video frames, the frame sequence number and the updated vehicle state feature.

[0036] Further, the behavior determination unit comprises:

[0037] extracting vehicle region coordinates of the same vehicle in two adjacent video frames according to the vehicle trajectory information, calculating an instantaneous speed according to the vehicle region coordinates, and determining that the vehicle is speeding when the instantaneous speed exceeds a preset threshold value;

[0038] dividing the monitoring area into a normal driving area and a vehicle prohibited area, and determining that the vehicle trajectory is abnormal when the vehicle region coordinates appear in the vehicle prohibited area;

[0039] calculating a vehicle position change according to the vehicle region coordinates of the same vehicle in two adjacent video frames, determining that the vehicle is in a stationary state when the vehicle position change is less than a preset change threshold value, calculating a continuous stationary duration of the vehicle, and determining that the vehicle is abnormally stationary when the continuous stationary duration of the vehicle exceeds a preset stationary threshold value;

[0040] calculating an angle difference between a vehicle forward direction and an allowed direction of an entrance and exit, and determining that the vehicle direction is in violation when the angle difference is greater than a threshold value;

[0041] warning for each detected abnormal behavior, and storing corresponding vehicle information and abnormal behavior information; the vehicle information includes a vehicle tracking ID, a vehicle type and a license plate number; the abnormal behavior information includes an abnormal behavior type, an abnormal behavior timestamp and a corresponding video segment index.

[0042] Compared with the prior art, the present application has the following advantages:

[0043] 1. The multi-modal video signal acquisition module acquires multi-modal video signals and environmental data by integrating visible light, infrared, thermal imaging cameras and environmental sensors. These signals are synchronized by time stamp to generate a multi-modal video frame sequence, and an environmental state vector is constructed based on data such as light intensity, temperature, precipitation and visibility. By combining the spatio-temporal consistency analysis of multi-source data, a basis is provided for video monitoring under complex weather conditions. The integrated multi-modal acquisition can more accurately capture scene features and significantly improve the adaptability of the system. The problem of insufficient perception ability of single video signal under low light, haze and rainfall conditions is solved, and the reliability of video data acquisition under complex weather conditions is enhanced.

[0044] 2、The environmental adaptability signal enhancement module selects the optimal strategy subset from the signal enhancement strategy set according to the environmental state vector. Through a machine learning algorithm, the module dynamically predicts and evaluates the effect of the enhancement strategy on the video frame, ensures the optimal signal enhancement, enhances the definition, contrast and noise reduction capability of the video image, in addition, the combination of the license plate color recognition subunit and the area relocation unit further improves the target detection and recognition effect of the new energy vehicle, especially in rain, snow, fog and other bad weather. Through the adaptability optimization of signal enhancement, the accuracy of the subsequent feature extraction and target detection module is effectively improved, which is beneficial to the monitoring task under complex weather conditions.

[0045] 3、The environmental adaptability detection and recognition module and the abnormal behavior monitoring module jointly constitute the intelligent analysis core of the system. Among them, the feature fusion unit generates a fusion feature vector in combination with the environmental state vector, which is used for target detection and recognition; the abnormal behavior monitoring module analyzes the abnormal behavior in combination with the vehicle trajectory information. The combination of detection and monitoring enables the system to locate and identify vehicles and their abnormal behaviors in real time. The feature fusion unit performs weight distribution and fusion on multi-modal data, effectively improving the robustness of target detection; the behavior monitoring module optimizes vehicle trajectory tracking through the Hungarian algorithm and Kalman filtering, further realizing accurate determination of behaviors such as overspeed and trajectory anomaly. Through fine-grained analysis and alarm strategy, the reliability of the video monitoring system under complex weather conditions is improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] Fig. 1 A structural schematic diagram of a video monitoring system based on image feature analysis according to the present application;

[0047] Fig. 2 A flowchart of a video monitoring system based on image feature analysis according to the present application;

[0048] Fig. 3 A flowchart of selecting an optimal signal enhancement strategy subset according to the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0050] Please refer to Figs. 1 to 3 The present application provides a video monitoring system based on image feature analysis, and the technical solutions are as follows:

[0051] Embodiment one:

[0052] A traffic hub in the core business district of a certain large city, the area has heavy traffic flow, complex traffic conditions, involves multiple trunk roads and mall parking lots, and has extremely high requirements for vehicle behavior monitoring and abnormal event identification. In order to realize accurate monitoring of large-scale traffic flow, a video monitoring system based on image feature analysis is applied, as shown in Fig. 1 The system includes a multi-modal video signal acquisition module, an environment adaptability signal enhancement module, an environment adaptability detection and identification module, and an abnormal behavior monitoring module. The environment adaptability detection and identification module includes a feature fusion unit, a target detection unit, and a target identification unit. The abnormal behavior monitoring module includes a monitoring and tracking unit and a behavior judgment unit. As shown in Fig. 2 The specific implementation steps include: S10. Collecting multi-modal video signals by integrating visible light video signals, infrared video signals, and thermal imaging video signals, and collecting environmental data; S20. Time stamp synchronization of the multi-modal video signals to generate a multi-modal video frame sequence, and constructing an environment state vector based on the environmental data; S30. Constructing a signal enhancement strategy set, selecting an optimal signal enhancement strategy subset from the signal enhancement strategy set based on the environment state vector; S40. Processing the multi-modal video frames through the optimal signal enhancement strategy subset to obtain optimized multi-modal video frames; S50. Extracting multi-modal signal features based on the optimized multi-modal video frames, and generating a fusion feature vector in combination with the environment state vector; S60. Extracting vehicle regions and license plate regions based on the fusion feature vector through a target detection network; S70. Identifying vehicle types and license plate numbers according to the vehicle regions and license plate regions; S80. Tracking vehicles in the multi-modal video frame sequence to obtain vehicle trajectory information; S90. Identifying abnormal behavior according to the vehicle trajectory information and alarming, and storing corresponding vehicle information and abnormal behavior information.

[0053] Further, the environmental data includes light intensity, temperature, precipitation, and visibility, which are collected by light intensity sensors, temperature sensors, precipitation sensors, and visibility sensors. Based on the environmental data, the system can dynamically adjust the signal enhancement strategy to further adapt to diverse weather conditions.

[0054] Further, the environment state vector in step S20 is obtained by normalizing the environmental data.

[0055] Further, in this embodiment, the signal enhancement strategy set includes a rain removal model, a defogging model, and an illumination compensation model. The rain removal model removes raindrops and rain streaks in the multi-modal video frames through a deep residual network. The defogging model restores the clarity of the multi-modal video frames through a dark channel defogging algorithm. The illumination compensation model adjusts the contrast and brightness of different regions of the multi-modal video frames based on adaptive histogram equalization.

[0056] Further, in the step S30, as shown in Fig. 3 selecting an optimal signal enhancement strategy subset from the signal enhancement strategy set based on the environment state vector comprises:

[0057] calculating contrast, definition and noise level according to the multi-modal video frame, to obtain an original quality vector;

[0058] predicting a predicted enhancement quality vector obtained after each signal enhancement strategy processes the multi-modal video frame according to the original quality vector and the environment state vector; the predicted enhancement quality vector comprises predicted contrast, predicted definition and predicted noise level;

[0059] calculating an enhancement strategy score according to the predicted enhancement quality vector:

[0060]

[0061] wherein score(p i , I, E) represents the enhancement strategy score; p i represents the i-th signal enhancement strategy in the signal enhancement strategy set; I represents the multi-modal video frame; E represents the environment state vector; represents the predicted contrast of using p i to process I; represents the predicted definition; represents the predicted noise level; R(p i , E) represents the correlation of the signal enhancement strategy p i and the environment state vector; w1, w2, w3 and w4 represent weight parameters; n represents the number of components in the environment state vector; β j represents the weight parameter of the j-th environment state component; G represents an environment strategy mapping matrix, and each element in the matrix represents the applicability of a signal enhancement strategy to an environment state component; G ij represents the applicability of p i to the j-th environment state component.

[0062] sorting all the signal enhancement strategies from large to small according to the enhancement strategy score, and selecting the first m signal enhancement strategies as the optimal signal enhancement strategy subset of the multi-modal video frame.

[0063] By calculating the original quality vector and the predicted enhancement quality vector, a mechanism for dynamically selecting an optimal signal enhancement strategy is provided, which effectively improves the pertinence of the enhancement strategy, reduces calculation redundancy, and ensures the real-time response capability of the system in complex weather environments.

[0064] Table 1 Signal enhancement effect evaluation under different weather conditions

[0065]

[0066] Table 1 shows the signal enhancement effect under different weather conditions. Through the environmental adaptability signal enhancement strategy, the system can significantly improve the image quality under various severe weather conditions, especially in extreme conditions such as heavy rain and thick fog, the vehicle license plate recognition accuracy is more obvious.

[0067] Further, the environmental complexity is calculated according to the environmental state vector, when the environmental complexity is greater than the maximum complexity threshold, the value of m is the number of signal enhancement strategies in the signal enhancement strategy set; when the environmental complexity is less than the minimum complexity threshold, the value of m is 1.

[0068] The calculation of the environmental complexity is the weighted sum of each component of the environmental state vector.

[0069] By calculating the dynamic adjustment of the optimal signal enhancement strategy number through the environmental complexity, it adapts to different complexity scenes, which not only ensures the enhancement effect, but also optimizes the utilization rate of computing resources.

[0070] Further, different weather environment data and image sample data under corresponding weather environment are collected to construct a training set; for each signal enhancement strategy, an enhancement prediction model is established through a machine learning method and trained through the training set; the original quality vector and the environmental state vector are input into the enhancement prediction model to obtain the predicted enhancement quality vector of each signal enhancement strategy.

[0071] Through machine learning method to train the enhancement prediction model, so that the signal enhancement process has self-learning ability, which enhances the adaptability of the system to unknown environment.

[0072] Further, the step S50 comprises:

[0073] According to the same time, the optimal multi-modal video frame is extracted to obtain the multi-modal signal feature;

[0074] According to the multi-modal signal feature and the environmental state vector, the multi-modal signal feature fusion is carried out to obtain the fusion feature vector; the formula of the multi-modal signal feature fusion is:

[0075]

[0076] Wherein, F fusion represents the fusion feature vector; M represents the number of multi-modal signal features; σ() represents the activation function; W i represents the weight matrix; E represents the environmental state vector; b idenotes a bias term; W i，s denotes a spatial attention weight matrix; W i，c denotes a channel attention weight matrix; F i denotes the i-th multi-modal signal feature.

[0077] The spatial attention weight matrix and the channel attention weight matrix include: performing maximum pooling and average pooling operations on the input multi-modal signal feature in the spatial dimension, concatenating the pooling results, and generating the spatial attention weight matrix through a 7x7 convolution layer and an activation function; at the same time, performing global average pooling on the multi-modal signal feature in the channel dimension, and generating the channel attention weight matrix through two perception machines.

[0078] The multi-modal signal feature fusion method combines the environmental state vector and the attention mechanism, can improve the accuracy and efficiency of target feature extraction, and provides more reliable data support for target detection and recognition.

[0079] Further, the step S60 and the step S70 include:

[0080] The target detection unit generates the vehicle region by performing target detection on the vehicle region using a Mask R-CNN model based on the fusion feature vector; and generates the license plate region by performing target detection on the vehicle region after extracting features from the vehicle region.

[0081] The target recognition unit extracts visual features of the vehicle region using a deep convolutional neural network, classifies the vehicle type through a multi-layer perception machine, performs character recognition on the license plate region using a Transformer model, and obtains the license plate number.

[0082] The introduction of Mask R-CNN and Transformer model not only improves the accuracy of vehicle region and license plate region detection, but also optimizes the license plate character recognition effect, ensuring the comprehensiveness and accuracy of target recognition.

[0083] Table 2 Multi-modal signal feature fusion effect evaluation

[0084]

[0085] Table 2 shows the multi-modal signal feature fusion effect of the system at different times within 24 hours. By adaptively adjusting the feature weights of different modalities, the system can maintain high vehicle classification accuracy under various lighting conditions.

[0086] Further, the specific implementation step further comprises a region relocation step: constructing an edge signal enhancement strategy set, selecting an optimal edge signal enhancement strategy subset from the edge signal enhancement strategy set based on the environment state vector, processing the optimized multi-modal video frame through the optimal edge signal enhancement strategy subset to obtain an updated optimized multi-modal video frame, and generating a relocated vehicle region and a relocated license plate region based on the updated optimized multi-modal video frame through steps S50 and S60.

[0087] Further, selecting an optimal edge signal enhancement strategy subset from the edge signal enhancement strategy set specifically comprises: setting the applicability of each edge signal enhancement strategy to the environment state component, sorting all edge signal enhancement strategies from large to small according to the weighted sum result of the corresponding applicability of all environment state components, and selecting the first m' edge signal enhancement strategies as the optimal edge signal enhancement strategy subset, wherein when the environment complexity is greater than the maximum complexity threshold, the value of m' is the number of edge signal enhancement strategies in the edge signal enhancement strategy set, and when the environment complexity is less than the minimum complexity threshold, the value of m' is 1.

[0088] The region relocation subunit improves the detection accuracy of the vehicle region in a targeted manner in combination with edge enhancement, thereby providing a guarantee for accurate identification and classification.

[0089] Further, the step S70 further comprises: performing color feature analysis on the license plate region, extracting color space features, calculating the green component proportion, and determining that the vehicle is a new energy vehicle when the green component proportion is greater than a preset green threshold, generating a relocated vehicle region to replace the vehicle region and generating a relocated license plate region to replace the license plate region through the region relocation step.

[0090] Since new energy vehicles are mostly designed with streamlined appearance, the contour features are not obvious under complex weather conditions, and through color analysis and dynamic region replacement, the accuracy of new energy vehicle monitoring is improved.

[0091] Further, the step S80 comprises:

[0092] extracting information from the vehicle region of each video frame to obtain a frame number and a vehicle state feature; the vehicle state feature comprises vehicle region coordinates, vehicle region size, and speed information;

[0093] predicting a predicted vehicle state feature in the current video frame through Kalman filtering based on the vehicle state feature of the previous video frame;

[0094] The prediction process is:

[0095] x k|k-1 = A x k-1|k-1 ;

[0096] P k|k-1 = A·P k-1|k-1 ·A T + Q;

[0097] wherein x k|k-1 represents a predicted vehicle state feature of a current video frame obtained according to a vehicle state feature of a previous video frame; x k-1|k-1 represents the vehicle state feature of the previous video frame; A represents a state transition matrix, A T represents a transpose of A; P k|k-1 represents a predicted covariance matrix of the current video frame; P k-1|k-1 represents a covariance matrix of the previous video frame; and Q represents a state transition noise covariance matrix;

[0098] According to the vehicle state feature of the current video frame and the predicted vehicle state feature, data association is performed by using a Hungarian algorithm to obtain a data association result; the specific operation of data association is as follows: a similarity matrix between the vehicle state feature of the current video frame and the predicted vehicle state feature is calculated, the Hungarian algorithm is used to match the similarity matrix, and an optimal matching result of the vehicle region and the predicted vehicle state feature is found.

[0099] The vehicle state feature of the current video frame is updated according to the data association result to obtain the vehicle trajectory information; the vehicle trajectory information includes a vehicle tracking ID of each vehicle in all video frames, the frame sequence number and the updated vehicle state feature.

[0100] The updating process is as follows:

[0101] K k = P k|k-1 ·H T ·(H·P k|k-1 ·H T + Q′) -1 ;

[0102] x k|k = x k|k-1 + K k ·(Z k -H·x k|k-1 );

[0103] P k|k = (I-K k ·H)·P k|k-1 ;

[0104] wherein K k represents a Kalman gain; H represents an observation matrix; H T represents a transpose of H; Q′ represents an observation noise covariance matrix; x k|kZ represents the vehicle state features of the current video frame after the update. k This represents the vehicle area observation value in the current video frame; P k|k This represents the covariance matrix of the current video frame after the update.

[0105] By extracting vehicle trajectory information, predicting with Kalman filter, and associating data with the Hungarian algorithm, high-precision vehicle tracking was achieved, laying a solid foundation for subsequent abnormal behavior analysis.

[0106] Further, step S90 includes:

[0107] The vehicle area coordinates of the same vehicle in two adjacent video frames are extracted based on the vehicle trajectory information. The instantaneous speed is calculated based on the vehicle area coordinates. When the instantaneous speed exceeds a preset threshold, it is determined to be speeding.

[0108] The monitoring area is divided into a normal driving area and a vehicle restricted area. When the vehicle area coordinates appear in the vehicle restricted area, it is determined to be an abnormal trajectory. The specific determination method is to use the ray method to determine whether the vehicle area coordinates appear in the vehicle restricted area.

[0109] The vehicle position change is calculated based on the vehicle area coordinates of the same vehicle in two adjacent video frames. When the vehicle position change is less than a preset change threshold, the vehicle is determined to be in a stationary state. The continuous stationary duration of the vehicle is calculated. When the continuous stationary duration of the vehicle exceeds a preset stationary threshold, it is determined to be an abnormal stationary state.

[0110] Calculate the angle difference between the vehicle's forward direction and the permitted direction at the entrance / exit. If the angle difference exceeds a threshold, it is determined to be a directional violation. The formula for calculating the angle difference is:

[0111] Δθ=min(|θ k -θ allowed |,2·π-|θ k -θ allowed |);

[0112] θ k =arctan((y′) k -y′ k-1 ) / (x′ k -x′ k-1 ));

[0113] Where Δθ represents the angle difference; min() represents the minimum value function; θ k θ represents the driving direction angle of the current video frame. allowed Indicates the permissible driving direction angle; (x′) k y′ k (x′) represents the center coordinates of the vehicle region in the current video frame; k-1, y' k-1 ) represents the vehicle region center coordinates of the previous video frame; arctan() represents the inverse tangent function;

[0114] For each detected abnormal behavior, an alarm is sounded, and the corresponding vehicle information and abnormal behavior information are stored; the vehicle information includes vehicle tracking ID, vehicle type and license plate number; the abnormal behavior information includes abnormal behavior type, abnormal behavior timestamp and corresponding video segment index.

[0115] The behavior determination unit designs a variety of abnormal behavior analysis mechanisms and is equipped with an intelligent alarm function, which improves the monitoring ability of the system to abnormal conditions and reduces potential safety hazards.

[0116] The video monitoring system based on image feature analysis provided by the application realizes efficient monitoring under complex weather conditions through the cooperative work of the multi-modal video signal acquisition module, the environment adaptability signal enhancement module, the environment adaptability detection and identification module and the abnormal behavior monitoring module. By integrating visible light, infrared, thermal imaging camera and environment sensor, the system can collect multi-modal data, and dynamically adapt the signal enhancement strategy according to the environment state vector, effectively solving the problem of decreased monitoring picture quality caused by bad weather. At the same time, the feature fusion unit combines multi-modal features and environment state vectors to improve the accuracy of target detection and identification. In addition, the abnormal behavior monitoring module realizes real-time analysis and processing of behaviors such as overspeed, abnormal trajectory and direction violation through trajectory tracking, behavior judgment and intelligent alarm. The application reduces the false positive rate and the false negative rate while ensuring the real-time and accuracy of the video monitoring system, improves the stability and robustness of the system, and improves the monitoring ability under complex weather conditions.

[0117] Embodiment two:

[0118] The system proposed by the application will be described again in the embodiments of the application. A video monitoring system based on image feature analysis, comprising:

[0119] The multi-modal video signal acquisition module collects multi-modal video signals by integrating visible light video signals, infrared video signals and thermal imaging video signals, simultaneously collects environment data, synchronizes the time stamps of the multi-modal video signals to generate a multi-modal video frame sequence, and constructs an environment state vector based on the environment data;

[0120] The environment adaptability signal enhancement module constructs a signal enhancement strategy set, selects an optimal signal enhancement strategy subset from the signal enhancement strategy set based on the environment state vector, and processes the multi-modal video frames through the optimal signal enhancement strategy subset to obtain optimized multi-modal video frames;

[0121] An environmental adaptability detection and recognition module includes a feature fusion unit, a target detection unit, and a target recognition unit. The feature fusion unit extracts multimodal signal features based on the optimized multimodal video frames and generates a fused feature vector by combining it with the environmental state vector. The target detection unit extracts vehicle regions and license plate regions based on the fused feature vector through a target detection network. The target recognition unit identifies the vehicle type and license plate number based on the vehicle regions and license plate regions.

[0122] The abnormal behavior monitoring module includes a monitoring and tracking unit and a behavior determination unit; the monitoring and tracking unit tracks vehicles in the multimodal video frame sequence to obtain vehicle trajectory information; the behavior determination unit identifies abnormal behavior based on the vehicle trajectory information, issues an alarm, and stores the corresponding vehicle information and abnormal behavior information.

[0123] Further, selecting the optimal subset of signal enhancement strategies from the set of signal enhancement strategies based on the environmental state vector includes:

[0124] The contrast, sharpness, and noise level are calculated based on the multimodal video frames to obtain the original quality vector;

[0125] Based on the original quality vector and the environmental state vector, a predicted enhanced quality vector is obtained after each signal enhancement strategy processes the multimodal video frame; the predicted enhanced quality vector includes predicted contrast, predicted sharpness, and predicted noise level.

[0126] Calculate the enhancement strategy score based on the predicted enhanced quality vector:

[0127]

[0128] Among them, score(p i (I, E) represents the score of the enhancement strategy; p i The signal enhancement strategy is represented by the i-th signal enhancement strategy in the set of signal enhancement strategies; I represents the multimodal video frame; E represents the environmental state vector. Indicates that p i The predicted contrast used to process I; This indicates the accuracy of the prediction; R(p) represents the predicted noise level; i E) represents the signal enhancement strategy p i Correlation with the environmental state vector; w1, w2, w3, and w4 represent weight parameters;

[0129] The signal enhancement strategies are ranked from large to small according to the enhancement strategy scores, and the top m signal enhancement strategies are selected as the optimal signal enhancement strategy subset of the multi-modal video frame.

[0130] Further, an environment complexity is calculated according to the environment state vector, when the environment complexity is greater than a maximum complexity threshold, the value of m is the number of signal enhancement strategies in the signal enhancement strategy set; when the environment complexity is less than a minimum complexity threshold, the value of m is 1.

[0131] Further, different weather environment data and image sample data under corresponding weather environment are collected to construct a training set; for each signal enhancement strategy, an enhancement prediction model is established by a machine learning method and trained by the training set; the original quality vector and the environment state vector are input into the enhancement prediction model to obtain the predicted enhancement quality vector of each signal enhancement strategy.

[0132] Further, the feature fusion unit comprises:

[0133] Multi-modal signal features are extracted according to the optimized multi-modal video frame at the same time;

[0134] Multi-modal signal feature fusion is performed according to the multi-modal signal features and the environment state vector to obtain the fusion feature vector; the formula of the multi-modal signal feature fusion is:

[0135]

[0136] Wherein, F fusion represents the fusion feature vector; M represents the number of multi-modal signal features; σ() represents an activation function; W i represents a weight matrix; E represents the environment state vector; b i represents a bias term; W i，s represents a spatial attention weight matrix; W i，c represents a channel attention weight matrix; F i represents the i-th multi-modal signal feature.

[0137] Further, the target detection unit and the target recognition unit comprise:

[0138] The target detection unit performs target detection based on the fusion feature vector using a Mask R-CNN model to generate the vehicle region; and after extracting features from the vehicle region, target detection is performed again to generate the license plate region;

[0139] The target recognition unit extracts visual features of the vehicle region by using a deep convolutional neural network, and classifies the vehicle type by using a multi-layer perceptron; and a character recognition is performed on the license plate region by using a Transformer model to obtain the license plate number.

[0140] Further, the environment adaptability detection and recognition module further comprises a region relocation unit, an edge signal enhancement strategy set is constructed, an optimal edge signal enhancement strategy subset is selected from the edge signal enhancement strategy set based on an environment state vector, and an updated optimized multi-modal video frame is obtained by processing the optimized multi-modal video frame through the optimal edge signal enhancement strategy subset; and a relocated vehicle region and a relocated license plate region are generated by the feature fusion unit and the target detection unit based on the updated optimized multi-modal video frame.

[0141] Further, the target recognition unit further comprises a license plate color recognition subunit, color feature analysis is performed on the license plate region, color space features are extracted, and a green component proportion is calculated; when the green component proportion is greater than a preset green threshold, the vehicle is determined to be a new energy vehicle, a relocated vehicle region is generated by the region relocation unit to replace the vehicle region, and a relocated license plate region is generated to replace the license plate region.

[0142] Further, the monitoring and tracking unit comprises:

[0143] Information extraction is performed on the vehicle region of each video frame to obtain a frame sequence number and a vehicle state feature; the vehicle state feature comprises a vehicle region coordinate, a vehicle region size, and speed information;

[0144] According to the vehicle state feature of the previous video frame, a predicted vehicle state feature in the current video frame is predicted by Kalman filtering;

[0145] According to the vehicle state feature of the current video frame and the predicted vehicle state feature, data association is performed by using the Hungarian algorithm to obtain a data association result;

[0146] The vehicle state feature of the current video frame is updated according to the data association result to obtain the vehicle trajectory information; the vehicle trajectory information comprises a vehicle tracking ID of each vehicle, the frame sequence number, and the updated vehicle state feature in all video frames.

[0147] Table 3 Abnormal behavior recognition performance evaluation

[0148]

[0149] Table 3 shows the performance of the system in identifying various abnormal behaviors. The system maintains a high identification accuracy while achieving low false positive and false negative rates, with an average response time of less than 1.2 seconds.

[0150] Further, the behavior determination unit comprises:

[0151] According to the vehicle trajectory information, the vehicle region coordinates of the same vehicle in two adjacent video frames are extracted, the instantaneous speed is calculated according to the vehicle region coordinates, and when the instantaneous speed exceeds a preset threshold, it is determined as speeding behavior;

[0152] The monitoring area is divided into a normal driving area and a vehicle prohibited area, and when the vehicle region coordinates appear in the vehicle prohibited area, it is determined as trajectory anomaly;

[0153] According to the vehicle region coordinates of the same vehicle in two adjacent video frames, the vehicle position change is calculated, when the vehicle position change is less than a preset change threshold, it is determined that the vehicle is in a stay state, the continuous stay duration of the vehicle is calculated, and when the continuous stay duration of the vehicle exceeds a preset stay threshold, it is determined as abnormal stay;

[0154] The angle difference between the vehicle forward direction and the entrance and exit allowed direction is calculated, and when the angle difference is greater than a threshold, it is determined as direction violation;

[0155] For each detected abnormal behavior, an alarm is given, and the corresponding vehicle information and abnormal behavior information are stored; the vehicle information includes vehicle tracking ID, vehicle type and license plate number; the abnormal behavior information includes abnormal behavior type, abnormal behavior timestamp and corresponding video segment index.

[0156] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A video surveillance system based on image feature analysis, characterized in that, include: The multimodal video signal acquisition module acquires multimodal video signals by integrating visible light video signals, infrared video signals, and thermal imaging video signals, while simultaneously acquiring environmental data, including light intensity, temperature, precipitation, and visibility; it timestamps the multimodal video signals to synchronously generate a multimodal video frame sequence, and constructs an environmental state vector based on the environmental data; An environmentally adaptive signal enhancement module constructs a signal enhancement strategy set, which includes a rain removal model, a fog removal model, and an illumination compensation model. Based on the environmental state vector, it selects an optimal subset of signal enhancement strategies from the set and processes the multimodal video frames using the optimal subset of signal enhancement strategies to obtain optimized multimodal video frames. Selecting an optimal subset of signal enhancement strategies from the set of signal enhancement strategies based on the environmental state vector includes: The contrast, sharpness, and noise level are calculated based on the multimodal video frames to obtain the original quality vector; Based on the original quality vector and the environmental state vector, a predicted enhanced quality vector is obtained after each signal enhancement strategy processes the multimodal video frame; the predicted enhanced quality vector includes predicted contrast, predicted sharpness, and predicted noise level. Calculate the enhancement strategy score based on the predicted enhanced quality vector: ； ； in, This represents the score of the enhancement strategy; This represents the i-th signal enhancement strategy in the set of signal enhancement strategies; This refers to the multimodal video frame; This represents the environmental state vector; Indicates will Used for processing The predicted contrast; Indicates will Used for processing The predicted clarity; Indicates will Used for processing The predicted noise level; This indicates the signal enhancement strategy. Correlation with the environmental state vector; , , and βj represents the weight parameter; n represents the number of components in the environmental state vector; βj represents the weight parameter of the j-th environmental state component; G represents the environmental policy mapping matrix, where each element represents the applicability of the signal enhancement policy to the environmental state component; Gij represents the applicability of pi to the j-th environmental state component. A training set is constructed based on collected data from different weather environments and corresponding image samples. For each signal enhancement strategy, an enhancement prediction model is established using machine learning methods and trained using the training set. The original quality vector and the environmental state vector are input into the enhancement prediction model to obtain the predicted enhancement quality vector for each signal enhancement strategy. The enhancement strategy scores are then sorted from largest to smallest, and the top-scoring strategies are selected. The signal enhancement strategies are a subset of the optimal signal enhancement strategies for the multimodal video frames; An environmental adaptability detection and recognition module includes a feature fusion unit, a target detection unit, and a target recognition unit. The feature fusion unit extracts multimodal signal features based on the optimized multimodal video frames and generates a fused feature vector by combining it with the environmental state vector. The target detection unit extracts vehicle regions and license plate regions based on the fused feature vector through a target detection network. The target recognition unit identifies the vehicle type and license plate number based on the vehicle regions and license plate regions. The abnormal behavior monitoring module includes a monitoring and tracking unit and a behavior determination unit; the monitoring and tracking unit tracks vehicles in the multimodal video frame sequence to obtain vehicle trajectory information; the behavior determination unit identifies abnormal behavior based on the vehicle trajectory information, issues an alarm, and stores the corresponding vehicle information and abnormal behavior information.

2. The video surveillance system based on image feature analysis according to claim 1, characterized in that, The environment complexity is calculated based on the environment state vector. When the environment complexity exceeds the maximum complexity threshold, ... The value is the number of signal enhancement strategies in the signal enhancement strategy set; When the environmental complexity is less than the minimum complexity threshold The value of is 1.

3. A video surveillance system based on image feature analysis according to claim 1, characterized in that, The feature fusion unit includes: Extract the multimodal signal features based on the optimized multimodal video frames at the same time. Multimodal signal feature fusion is performed based on the multimodal signal features and the environmental state vector to obtain the fused feature vector; the formula for multimodal signal feature fusion is: ； in, This represents the fused feature vector; This indicates the number of features in the multimodal signal; Indicates the activation function; Represents the weight matrix; This represents the environmental state vector; Indicates the bias term; Represents the spatial attention weight matrix; This represents the channel attention weight matrix; This represents the i-th feature of the multimodal signal.

4. A video surveillance system based on image feature analysis according to claim 1, characterized in that, The target detection unit and the target recognition unit include: The target detection unit uses the Mask R-CNN model to perform target detection based on the fused feature vector to generate the vehicle region; after extracting features from the vehicle region, it performs target detection again to generate the license plate region. The target recognition unit uses a deep convolutional neural network to extract visual features of the vehicle area, classifies the vehicle type using a multilayer perceptron, and uses a Transformer model to perform character recognition on the license plate area to obtain the license plate number.

5. A video surveillance system based on image feature analysis according to claim 1, characterized in that, The environmental adaptability detection and recognition module further includes a region relocation unit, which constructs an edge signal enhancement strategy set, selects an optimal subset of edge signal enhancement strategies from the edge signal enhancement strategy set based on the environmental state vector, processes the optimized multimodal video frame using the optimal subset of edge signal enhancement strategies to obtain an updated optimized multimodal video frame, and generates a relocated vehicle region and a relocated license plate region based on the updated optimized multimodal video frame using the feature fusion unit and the target detection unit.

6. A video surveillance system based on image feature analysis according to claim 4, characterized in that, The target recognition unit also includes a license plate color recognition subunit, which performs color feature analysis on the license plate area, extracts color space features, calculates the green component ratio, and determines that the vehicle is a new energy vehicle when the green component ratio is greater than a preset green threshold. The repositioning unit generates a repositioned vehicle area to replace the vehicle area and generates a repositioned license plate area to replace the license plate area.

7. A video surveillance system based on image feature analysis according to claim 1, characterized in that, The monitoring and tracking unit includes: Information is extracted from the vehicle region of each video frame to obtain the frame number and vehicle state features; the vehicle state features include vehicle region coordinates, vehicle region size, and speed information. Based on the vehicle state features described in the previous video frame, predict the vehicle state features in the current video frame using Kalman filtering. Based on the vehicle state features and the predicted vehicle state features of the current video frame, the Hungarian algorithm is used to perform data association to obtain the data association result. The vehicle status features of the current video frame are updated based on the data association results to obtain the vehicle trajectory information; the vehicle trajectory information includes the vehicle tracking ID of each vehicle in all video frames, the frame number, and the updated vehicle status features.

8. A video surveillance system based on image feature analysis according to claim 1, characterized in that, The behavior determination unit includes: The vehicle area coordinates of the same vehicle in two adjacent video frames are extracted based on the vehicle trajectory information. The instantaneous speed is calculated based on the vehicle area coordinates. When the instantaneous speed exceeds a preset threshold, it is determined to be speeding. The monitoring area is divided into a normal driving area and a vehicle restricted area. When the coordinates of the vehicle area appear in the vehicle restricted area, it is determined to be an abnormal trajectory. The vehicle position change is calculated based on the vehicle area coordinates of the same vehicle in two adjacent video frames. When the vehicle position change is less than a preset change threshold, the vehicle is determined to be in a stationary state. The continuous stationary duration of the vehicle is calculated. When the continuous stationary duration of the vehicle exceeds a preset stationary threshold, it is determined to be an abnormal stationary state. Calculate the angle difference between the vehicle's forward direction and the permitted direction of the entrance / exit. When the angle difference is greater than a threshold, it is determined to be a directional violation. An alarm is triggered for each of the detected abnormal behaviors, and the corresponding vehicle information and abnormal behavior information are stored; the vehicle information includes the vehicle tracking ID, the vehicle type, and the license plate number; the abnormal behavior information includes the abnormal behavior type, the abnormal behavior timestamp, and the corresponding video segment index.

Images