Intelligent video traffic monitoring system based on multiple viewpoints
Through multi-viewpoint fusion processing and deep learning technology, occlusion problems in complex traffic environments are solved, the accuracy of target recognition and tracking is improved, data security and intelligent analysis capabilities are enhanced, and a comprehensive intelligent video traffic monitoring system is built.
Patent Information
- Application Number
- CN202411124877.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In complex traffic environments, the phenomenon of occlusion between vehicles and pedestrians makes it impossible for the monitoring system to accurately track or identify targets, and the existing technology is difficult to effectively solve.
Using a multi-viewpoint-based intelligent video traffic monitoring system, the multi-viewpoint fusion processing module is used to spatially align and fusion video images from different angles. Combined with deep learning technologies such as the Transformer model, the occlusion phenomenon is corrected and predicted to eliminate blind spots in the perspective. And LSTM and large-scale data sets are introduced for training of intelligent analysis models to improve the accuracy of target recognition and tracking.
Improve the accuracy and robustness of target recognition and tracking, strengthen data encryption and transmission security, build a more comprehensive intelligent analysis model, which can automatically learn and extract target features, ensure the security of video data during transmission and Data access compliance.
Smart Images

Figure CN118968781A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of traffic monitoring, and in particular to an intelligent video traffic monitoring system based on multiple viewpoints. Background Art
[0002] With the widespread use of high-definition cameras, infrared cameras and other equipment, as well as the continuous advancement of image processing technology, traffic monitoring systems can capture and analyze information in traffic scenes in real time, providing strong support for traffic management. Multi-viewpoint monitoring technology forms a multi-angle, all-round monitoring network by deploying multiple cameras in the same area.
[0003] After searching, the invention patent with Chinese patent number CN117319609A discloses an IoT big data intelligent video monitoring system and method, which belongs to the field of IoT and big data technology. It solves the problem that traditional video monitoring systems usually rely on manual operation and cannot perform advanced intelligent analysis, which leads to insufficient analysis and utilization of large-scale monitoring data. The storage and management costs of large-scale video data are high, and there are challenges for long-term storage and rapid retrieval. Traditional storage methods lack effective data management strategies. Traditional monitoring systems usually run alone, and it is difficult to associate and comprehensively analyze data from different monitoring points, and it is impossible to provide more comprehensive scene analysis; it includes video acquisition module, data storage module, data processing module, intelligent analysis module, and geographic location system integration module.
[0004] Compared with the existing technology, this invention patent with Chinese patent number CN117319609A captures traffic conditions on the road through cameras on traffic sections, including traffic flow, traffic accidents, traffic signal status, etc. The captured video and traffic data will be transmitted to the traffic management center in real time and stored in the distributed file system and cloud storage service in the system. The original video data will be processed through operations such as encoding and decoding, compression, image enhancement and feature extraction. The intelligent analysis module is used to perform intelligent analysis on video data, detect abnormal events, identify targets, generate alarms and notifications in real time, and generate statistical information and reports. The system displays monitoring points, event locations and traffic conditions on a map.
[0005] However, when the above device is used, in a complex traffic environment, occlusion between vehicles and pedestrians is a common phenomenon, which may cause the monitoring system to be unable to accurately track or identify the target. Therefore, an intelligent video traffic monitoring system based on multi-viewpoints is proposed. Summary of the invention
[0006] The purpose of the present invention is to solve the problem in the prior art that in a complex traffic environment, occlusion between vehicles and pedestrians is a common phenomenon, which may cause the monitoring system to be unable to accurately track or identify the target, and propose an intelligent video traffic monitoring system based on multi-viewpoints.
[0007] In order to achieve the above object, the present invention adopts the following technical solutions:
[0008] An intelligent video traffic monitoring system based on multiple viewpoints, comprising:
[0009] Video acquisition module: responsible for capturing video data in real time and transmitting the data to the multi-view fusion processing module through the Internet of Things protocol and communication standards;
[0010] Multi-viewpoint fusion processing module: responsible for spatial alignment and fusion of video images from multiple perspectives to eliminate blind spots;
[0011] Intelligent analysis module: responsible for building intelligent analysis models and conducting intelligent analysis;
[0012] Data storage and management module: responsible for redundant storage and fault recovery of video data;
[0013] Data encryption and security module: responsible for encrypting video data;
[0014] User interaction and alarm module: responsible for providing the user interface and sending alarm information to relevant personnel when the system detects abnormal situations (such as traffic congestion, violations, accidents, etc.);
[0015] System configuration and management module: responsible for the overall configuration and management of the system, including camera parameter settings, user authority management, system log recording, etc.;
[0016] The video data captured in real time by the video acquisition module is transmitted to the multi-view fusion processing module through the Internet of Things protocol and communication standard. The multi-view fusion processing module transmits the fused video data to the intelligent analysis module. The intelligent analysis module transmits the analysis results (such as traffic flow data, congestion conditions, violation records, etc.) to the data storage and management module for storage. At the same time, the intelligent analysis module requests the data storage and management module to provide historical data. When the intelligent analysis module detects an abnormal situation (such as traffic congestion, violation, accident, etc.), the intelligent analysis module sends the alarm information to the user interaction and alarm module. The user interaction module requests to view historical monitoring videos or reports. The data storage and management module is responsible for providing historical monitoring videos or reports.
[0017] The above technical solution further includes:
[0018] Furthermore, the video acquisition module includes a camera unit, a data acquisition and processing unit, a data transmission unit, and a control and management unit. The camera unit includes multiple high-definition cameras for capturing video data of traffic scenes. The camera has night vision, automatic focus, wide-angle and other functions to meet different traffic monitoring needs. The data acquisition and processing unit is responsible for receiving raw video data from the camera and performing preliminary processing, such as format conversion, noise suppression, etc. The data transmission unit uses the Internet of Things protocol (such as MQTT, CoAP, etc.) and communication standards (such as Wi-Fi, 4G / 5G, optical fiber, etc.) to transmit the processed video data to the multi-view fusion processing module in real time. The control and management unit is responsible for remote control and management of the camera, including camera switching, focal length adjustment, direction rotation, etc. At the same time, the control and management unit is responsible for monitoring the operating status of the camera, timely discovering and handling faults, the camera unit sends the captured video data of the traffic scene to the data acquisition and processing unit, the data acquisition and processing unit sends the processed video data to the data transmission unit, and the control and management unit sends control instructions to the camera unit, such as adjusting the focal length, direction, etc.
[0019] Furthermore, the multi-viewpoint fusion processing module includes a video receiving unit, an image preprocessing unit, an image registration unit, an image fusion unit, an occlusion correction and prediction unit, and a result output unit. The video receiving unit is responsible for receiving video data streams from cameras at different angles, performing preliminary analysis and verification on the received video data, and ensuring the integrity and accuracy of the data. The image preprocessing unit is responsible for preprocessing the received video frames, including denoising, enhancement, correction, etc., to improve the efficiency and accuracy of subsequent processing. The image registration unit is responsible for spatially aligning the video data streams from cameras at different angles using an image registration algorithm. The image fusion unit is responsible for fusing the video data from cameras at different angles on the basis of image registration to eliminate blind spots in viewing angles and form a more complete view of the traffic scene. The occlusion correction and prediction unit is responsible for combining deep learning technology , correct and predict the occlusion phenomenon, learn the characteristics and movement laws of the occlusion object by training the deep learning model, and be able to infer the position and state of the occluded object when occlusion occurs. The result output unit is responsible for outputting the fused video image and the results of occlusion correction and prediction to the subsequent intelligent analysis module or user interaction module. The video receiving unit sends the received video data streams from cameras of different angles to the image preprocessing unit for preliminary processing. The image preprocessing unit sends the preprocessed video frame to the image registration unit for spatial alignment. The image registration unit sends the registered video image to the image fusion unit for fusion processing. During the fusion processing, the occlusion correction and prediction units work simultaneously to correct and predict the occlusion phenomenon in the fused image. The result output unit receives the video image after the fusion processing is completed and the occlusion phenomenon is corrected and predicted.
[0020] Furthermore, the occlusion correction and prediction unit uses Transformer to correct and predict the occlusion phenomenon, including the following steps:
[0021] Data preprocessing: Obtain images or video sequences containing occlusion phenomena, and perform data cleaning on the images or video sequences containing occlusion phenomena, including removing noise, adjusting image size, etc., and increase data diversity through operations such as rotation, scaling, and cropping to improve the generalization ability of the model;
[0022] Feature extraction:
[0023] Transformer input: convert the preprocessed image or video sequence into an input format acceptable to the Transformer model, such as a token sequence;
[0024] Positional encoding: Since the Transformer model itself does not have the ability to process the position information of elements in the sequence, it is necessary to add positional encoding to distinguish elements at different positions. The specific formula for adding positional encoding is as follows:
[0025]
[0026] Among them, PE (pos,i) Represents the encoded value at position pos, where pos is the position index of the current element in the sequence, and i is the dimension index, which is used to traverse the dimensions of the position encoding vector (counting starts from 0, but usually the dimension index is interpreted from 1 or a more specific starting value. Here, to be consistent with the formula, it is explained from 0). model is the dimension of the Transformer model, i.e. the dimension of the input embedding, which determines the vector length of the positional encoding;
[0027] Self-attention mechanism: The Transformer model extracts global features from an image or video sequence through a self-attention mechanism, which allows the model to take into account other elements in the sequence when processing each element, thereby capturing the dependencies between elements.
[0028] Occlusion detection and recognition:
[0029] Feature analysis: Analyze the extracted features and identify occlusions in images or video sequences, which involves the interpretation of feature maps and the application of classifiers;
[0030] Occlusion modeling: Model the detected occlusions and analyze their properties such as shape, size, and position;
[0031] Occlusion Correction and Prediction:
[0032] Occlusion correction:
[0033] Local feature fusion: The Transformer model is used to fuse local features before and after occlusion to restore the information of the occluded part. This is achieved by adjusting the attention weight so that the model pays more attention to the features around the occluded part.
[0034] Global feature alignment: Use global features to guide the correction of occluded parts to ensure that the corrected features are consistent with the overall image or video sequence;
[0035] predict:
[0036] Time series prediction: For video sequences, the sequence modeling capability of the Transformer model is used to predict the target state during occlusion.
[0037] Trajectory prediction: Based on the target trajectory and feature changes before occlusion, the target's motion trajectory and position during occlusion are predicted;
[0038] Loss function optimization: By defining appropriate loss functions (such as mean square error, cross entropy, etc.), the model parameters are optimized to achieve the best performance in occlusion correction and prediction.
[0039] Output results: Output the image or video sequence after occlusion correction, in which the occluded part is effectively restored, and output the target prediction results during the occlusion period, including position, speed, acceleration and other information.
[0040] Furthermore, the intelligent analysis module includes a data input unit, a preprocessing unit, a model training unit and an intelligent analysis unit. The data input unit is responsible for receiving the fused video data and historical data provided by the multi-viewpoint fusion processing module and the data storage and management module as input data. The input data includes video images, traffic flow data, road network structure information, weather conditions, etc. The preprocessing unit preprocesses the input data, such as format conversion, noise suppression, data cleaning, etc., to ensure data quality. The model training unit uses LSTM and large-scale data sets for training. The intelligent analysis unit combines traffic flow theory, road network structure and weather conditions multi-source information to construct an intelligent analysis model to realize traffic flow monitoring, congestion identification, violation detection, accident warning and other intelligent analysis functions. The data input unit sends the received input data to the preprocessing unit for cleaning and formatting. The preprocessing unit uses the preprocessed data as training data for the deep learning model, and the trained deep learning model is deployed in the intelligent analysis unit.
[0041] Furthermore, the model training unit uses LSTM and a large-scale data set for training, including the following steps:
[0042] Data preparation:
[0043] Collect the fused video data and historical data provided by the multi-view fusion processing module and the data storage and management module, extract the feature data related to the target task, and divide the data set into a training set, a validation set, and a test set, with a ratio of 70%, 15%, and 15%;
[0044] Model construction:
[0045] Building a model using LSTM:
[0046] Forget gate: f t =σ(W f ·[h t-1 ,x t ]+b f )
[0047] Among them, W f is the weight matrix of the forget gate, b f is the bias term, σ is the sigmoid function, [h t-1 ,x t ] means to set h t-1 and x t Concatenate into a vector;
[0048] Input gate: The input gate is responsible for updating the cell state. The input gate consists of two parts: one is the sigmoid layer, which determines which information will be updated; the other is the tanh layer, which creates a new candidate value vector. The new candidate value vector is added to the cell state. Finally, the two information are multiplied to update the cell state.
[0049] Sigmoid layer: i t =σ(W i ·[h t-1 ,x t ]+b i );
[0050] Tanh layer:
[0051] Cell status update:
[0052] Output gate: The output gate determines which part of the information based on the cell state is used for output, according to the current input tx t , the state of the hidden layer at the previous moment h t-1 And the latest cell state C t , through the interaction of sigmoid function and tanh function, the output h at the current moment is determined t ;
[0053] Sigmoid layer: o t =σ(W o ·[h t-1 ,x t ]+b o );
[0054] Output hidden state: h t =o t *tanh(C t );
[0055] Training process:
[0056] Forward propagation: For each time step of input, the hidden state and output are calculated according to the RNN;
[0057] Calculate loss: Use a loss function to measure the difference between the model prediction and the actual label;
[0058] Back propagation: Calculate the gradient of loss with respect to model parameters through the time back propagation algorithm;
[0059] Parameter update: Use Adam to update model parameters according to the gradient;
[0060] Testing and Reasoning:
[0061] The model performance is evaluated on the test set. During inference, the data to be classified is input into the trained RNN model. The model outputs the probability distribution of each category, and the category with the highest probability is taken as the classification result.
[0062] Furthermore, the data storage and management module includes a distributed file system, a cloud storage service unit, a data indexing and retrieval unit, a data backup unit and a fault recovery unit. The distributed file system is responsible for distributing and storing the video data on multiple nodes to achieve data redundancy and fault tolerance. Commonly used distributed file systems include GFS (Google File System), HDFS (Hadoop Distributed File System), etc. The cloud storage service unit uses the storage resources of cloud storage service providers (such as AWS, Azure, Alibaba Cloud, etc.) to store video data in the cloud. The data indexing and retrieval unit is responsible for establishing an index of the video data for fast retrieval and access. The index can be based on the metadata of the video (such as timestamp, location, event type, etc.) or the characteristics of the video content (such as images, sounds, etc.). The data backup unit is responsible for regularly backing up the video data to prevent data loss or damage. The backup can be stored locally or in the cloud, and can be configured as incremental backup or full backup. In the event of data loss, node failure or system crash, the fault recovery unit is responsible for recovering data from the backup. The fault recovery strategy may include data replication, data migration, data reconstruction, etc. The distributed file system will Part or all of the video data is stored in a cloud storage service unit to achieve remote backup and access to the data. The distributed file system and the cloud storage service unit perform data transmission and synchronization through an API interface or a standard protocol (such as HTTP, HTTPS). The data indexing and retrieval unit accesses the video data in the distributed file system or cloud storage service, establishes and maintains indexes. The data backup unit periodically reads video data from the distributed file system or cloud storage service unit and backs it up to a designated storage location. The backup process involves operations such as data compression, encryption, and transmission. When a system failure occurs and data is lost, the fault recovery unit triggers a data recovery process. The fault recovery unit obtains the backup data from the data backup unit and then restores it to the original location or a new storage location.
[0063] Furthermore, the data encryption and security module includes a data encryption unit, a secure communication unit, an access control and authority management unit, and a key management unit. The data encryption unit is responsible for encrypting the video data and converting the plaintext data into ciphertext data using an encryption algorithm (such as AES, RSA, etc.). The encryption process can be performed before the data is sent (transmission encryption) or when the data is stored (storage encryption). The secure communication unit is responsible for introducing end-to-end encryption technology and TLS / SSL protocol to ensure the security of data during transmission. End-to-end encryption means that the data is always encrypted when it is transmitted between the sender and the receiver, and the intermediate nodes cannot decrypt or view the data content. The TLS / SSL protocol encrypts the client and the server. The communication between the two parties, as well as the verification of the identity of the server, are used to ensure the security of data transmission. The access control and permission management unit is responsible for establishing a complete access control and permission management mechanism to strictly control and review the access to data. The access control and permission management unit will decide whether the user accesses specific video data based on the user's identity and permission. Permission management may include role management, policy management, audit logs and other functions to ensure the compliance and traceability of data access. The key management unit is responsible for the generation, storage, distribution, update and destruction of encryption keys and other life cycle management. The data encryption unit encrypts the video data and transmits it through the secure communication unit. The key management unit provides the data encryption unit with the key required for encryption.
[0064] The present invention has the following beneficial effects:
[0065] 1. In the present invention, multi-viewpoint fusion technology is introduced to fuse images captured by cameras at different angles to improve the accuracy of target recognition and tracking. At the same time, Transformer is used to correct and predict occlusion phenomena. The Transformer model can automatically learn and extract effective features of the target, thereby improving the accuracy and robustness of tracking.
[0066] 2. In the present invention, data encryption and transmission security are strengthened, and end-to-end encryption technology is used to ensure the security of video data during transmission. At the same time, a complete access control and authority management mechanism is established to prevent unauthorized access and data leakage.
[0067] 3. In the present invention, LSTM and large-scale data sets are introduced for training to improve the system's behavior understanding and prediction capabilities. At the same time, a more comprehensive intelligent analysis model is constructed by combining multi-source information such as traffic flow theory and road network structure. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] Figure 1 A system block diagram of a multi-viewpoint intelligent video traffic monitoring system proposed by the present invention;
[0069] Figure 2 It is a flow chart of the multi-view fusion processing module in the present invention. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0071] See also Figure 1-Figure 2 As shown, the present invention is an intelligent video traffic monitoring system based on multiple viewpoints, comprising:
[0072] Video acquisition module: responsible for capturing video data in real time and transmitting the data to the multi-view fusion processing module through the Internet of Things protocol and communication standards;
[0073] Multi-viewpoint fusion processing module: responsible for spatial alignment and fusion of video images from multiple perspectives to eliminate blind spots;
[0074] Intelligent analysis module: responsible for building intelligent analysis models and conducting intelligent analysis;
[0075] Data storage and management module: responsible for redundant storage and fault recovery of video data;
[0076] Data encryption and security module: responsible for encrypting video data;
[0077] User interaction and alarm module: responsible for providing the user interface and sending alarm information to relevant personnel when the system detects abnormal situations (such as traffic congestion, violations, accidents, etc.);
[0078] System configuration and management module: responsible for the overall configuration and management of the system, including camera parameter settings, user authority management, system log recording, etc.;
[0079] The video data captured in real time by the video acquisition module is transmitted to the multi-view fusion processing module through the Internet of Things protocol and communication standards. The multi-view fusion processing module transmits the fused video data to the intelligent analysis module. The intelligent analysis module transmits the analysis results (such as traffic flow data, congestion conditions, violation records, etc.) to the data storage and management module for storage. At the same time, the intelligent analysis module requests the data storage and management module to provide historical data. When the intelligent analysis module detects an abnormal situation (such as traffic congestion, violation, accident, etc.), the intelligent analysis module sends the alarm information to the user interaction and alarm module. The user interaction module requests to view historical monitoring videos or reports. The data storage and management module is responsible for providing historical monitoring videos or reports.
[0080] The working principle of the multi-viewpoint intelligent video traffic monitoring system proposed in the present invention is that the video acquisition module captures video data in real time through high-definition cameras deployed on major traffic arteries and intersections in the city, and the video acquisition module transmits the real-time captured video data to the multi-viewpoint fusion processing module through the Internet of Things protocol and communication standards;
[0081] The multi-view fusion processing module receives video data from cameras at different angles, processes it using image registration and fusion algorithms, eliminates blind spots, improves the accuracy of target recognition and tracking, and combines Transformer to correct and predict occlusion phenomena;
[0082] The intelligent analysis module receives the fused video data and uses deep learning models and algorithms to perform intelligent analysis to analyze traffic flow, congestion, violations, etc., predict potential traffic risks and accidents, and request the data storage and management module to provide historical data to support the analysis process. When it is detected that the traffic flow of a certain section or intersection exceeds the preset threshold, or the vehicle speed drops significantly, a traffic congestion alarm will be triggered. When multiple traffic violations are identified, such as running a red light, driving in the wrong direction, driving on the line, driving in the wrong lane, etc., a violation alarm will be generated, and the license plate number, violation time, violation location, and violation picture or video evidence of the violating vehicle will be attached. The alarm information will be passed to the user interaction and alarm module to notify relevant personnel;
[0083] The data storage and management module is responsible for storing the analysis results and historical video data generated by the intelligent analysis module, and uses a distributed file system and cloud storage service to ensure data reliability and accessibility.
[0084] In one embodiment, for the above-mentioned video acquisition module, the video acquisition module includes a camera unit, a data acquisition and processing unit, a data transmission unit, and a control and management unit. The camera unit includes multiple high-definition cameras for capturing video data of traffic scenes. The camera has night vision, automatic focus, wide-angle and other functions to meet different traffic monitoring needs. The data acquisition and processing unit is responsible for receiving raw video data from the camera and performing preliminary processing, such as format conversion, noise suppression, etc. The data transmission unit uses the Internet of Things protocol (such as MQTT, CoAP, etc.) and communication standards (such as Wi-Fi, 4G / 5G, optical fiber, etc.) to transmit the processed video data to the multi-view fusion processing module in real time. The control and management unit is responsible for remote control and management of the camera, including the camera's switch, focal length adjustment, direction rotation, etc. At the same time, the control and management unit is responsible for monitoring the camera's operating status, timely discovering and handling faults, the camera unit sends the captured video data of the traffic scene to the data acquisition and processing unit, the data acquisition and processing unit sends the processed video data to the data transmission unit, and the control and management unit sends control instructions to the camera unit, such as adjusting the focal length, direction, etc.
[0085] In one embodiment, for the above-mentioned multi-view fusion processing module, the multi-view fusion processing module includes a video receiving unit, an image preprocessing unit, an image registration unit, an image fusion unit, an occlusion correction and prediction unit, and a result output unit. The video receiving unit is responsible for receiving video data streams from cameras at different angles, performing preliminary analysis and verification on the received video data to ensure the integrity and accuracy of the data. The image preprocessing unit is responsible for preprocessing the received video frames, including denoising, enhancement, correction, etc., to improve the efficiency and accuracy of subsequent processing. The image registration unit is responsible for using the image registration algorithm to spatially align the video data streams from cameras at different angles. The image fusion unit is responsible for fusing the video data from cameras at different angles on the basis of image registration to eliminate blind spots in viewing angles and form a more complete view of the traffic scene. The occlusion correction and prediction unit is responsible for It is responsible for combining deep learning technology to correct and predict occlusion phenomena. By training deep learning models, it learns the characteristics and movement laws of occlusion objects, and can infer the position and state of the occluded object when occlusion occurs. The result output unit is responsible for outputting the fused video image and the results of occlusion correction and prediction to the subsequent intelligent analysis module or user interaction module. The video receiving unit sends the received video data streams from cameras at different angles to the image preprocessing unit for preliminary processing. The image preprocessing unit sends the preprocessed video frames to the image registration unit for spatial alignment. The image registration unit sends the registered video images to the image fusion unit for fusion processing. During the fusion processing, the occlusion correction and prediction units work simultaneously to correct and predict the occlusion phenomena in the fused image. The result output unit receives the video image after the fusion processing is completed and the occlusion phenomena are corrected and predicted.
[0086] In one embodiment, for the occlusion correction and prediction unit, the occlusion correction and prediction unit uses Transformer to correct and predict the occlusion phenomenon, including the following steps:
[0087] Data preprocessing: Obtain images or video sequences containing occlusion phenomena, and perform data cleaning on the images or video sequences containing occlusion phenomena, including removing noise, adjusting image size, etc., and increase data diversity through operations such as rotation, scaling, and cropping to improve the generalization ability of the model;
[0088] Feature extraction:
[0089] Transformer input: convert the preprocessed image or video sequence into an input format acceptable to the Transformer model, such as a token sequence;
[0090] Positional encoding: Since the Transformer model itself does not have the ability to process the position information of elements in the sequence, it is necessary to add positional encoding to distinguish elements at different positions. The specific formula for adding positional encoding is as follows:
[0091]
[0092]
[0093] Among them, PE (pos,i) Represents the encoded value at position pos, where pos is the position index of the current element in the sequence, and i is the dimension index, which is used to traverse the dimensions of the position encoding vector (counting starts from 0, but usually the dimension index is interpreted from 1 or a more specific starting value. Here, to be consistent with the formula, it is explained from 0). model is the dimension of the Transformer model, i.e. the dimension of the input embedding, which determines the vector length of the positional encoding;
[0094] Self-attention mechanism: The Transformer model extracts global features from images or video sequences through the self-attention mechanism. The self-attention mechanism allows the model to consider other elements in the sequence when processing each element, thereby capturing the dependencies between elements.
[0095] Occlusion detection and recognition:
[0096] Feature analysis: Analyze the extracted features and identify occlusions in images or video sequences, which involves the interpretation of feature maps and the application of classifiers;
[0097] Occlusion modeling: Model the detected occlusions and analyze their properties such as shape, size, and position;
[0098] Occlusion Correction and Prediction:
[0099] Occlusion correction:
[0100] Local feature fusion: The Transformer model is used to fuse local features before and after occlusion to restore the information of the occluded part. This is achieved by adjusting the attention weight so that the model pays more attention to the features around the occluded part.
[0101] Global feature alignment: Use global features to guide the correction of occluded parts to ensure that the corrected features are consistent with the overall image or video sequence;
[0102] predict:
[0103] Time series prediction: For video sequences, the sequence modeling capability of the Transformer model is used to predict the target state during occlusion.
[0104] Trajectory prediction: Based on the target trajectory and feature changes before occlusion, the target's motion trajectory and position during occlusion are predicted;
[0105] Loss function optimization: By defining appropriate loss functions (such as mean square error, cross entropy, etc.), the model parameters are optimized to achieve the best performance in occlusion correction and prediction.
[0106] Output results: Output the image or video sequence after occlusion correction, in which the occluded part is effectively restored, and output the target prediction results during the occlusion period, including position, speed, acceleration and other information.
[0107] In one embodiment, for the above-mentioned intelligent analysis module, the intelligent analysis module includes a data input unit, a preprocessing unit, a model training unit and an intelligent analysis unit. The data input unit is responsible for receiving the fused video data and historical data provided by the multi-viewpoint fusion processing module and the data storage and management module as input data. The input data includes video images, traffic flow data, road network structure information, weather conditions, etc. The preprocessing unit preprocesses the input data, such as format conversion, noise suppression, data cleaning, etc., to ensure data quality. The model training unit uses LSTM and large-scale data sets for training. The intelligent analysis unit combines traffic flow theory, road network structure and weather conditions multi-source information to build an intelligent analysis model to realize traffic flow monitoring, congestion identification, violation detection, accident warning and other intelligent analysis functions. The data input unit sends the received input data to the preprocessing unit for cleaning and formatting. The preprocessing unit uses the preprocessed data as training data for the deep learning model, and the trained deep learning model is deployed in the intelligent analysis unit.
[0108] In one embodiment, for the above-mentioned model training unit, the model training unit uses LSTM and a large-scale data set for training, including the following steps:
[0109] Data preparation:
[0110] Collect the fused video data and historical data provided by the multi-view fusion processing module and the data storage and management module, extract the feature data related to the target task, and divide the data set into a training set, a validation set, and a test set, with a ratio of 70%, 15%, and 15%;
[0111] Model construction:
[0112] Building a model using LSTM:
[0113] Forget gate: f t =σ(W f ·[h t-1 ,x t ]+b f )
[0114] Among them, W f is the weight matrix of the forget gate, b f is the bias term, σ is the sigmoid function, [h t-1 ,x t ] means to set h t-1 and x t Concatenate into a vector;
[0115] Input gate: The input gate is responsible for updating the cell state. The input gate consists of two parts: one is the sigmoid layer, which determines which information will be updated; the other is the tanh layer, which creates a new candidate value vector. The new candidate value vector is added to the cell state. Finally, the two information are multiplied to update the cell state.
[0116] Sigmoid layer: i t =σ(W i ·[h t-1 ,x t ]+b i );
[0117] Tanh layer:
[0118] Cell status update:
[0119] Output gate: The output gate determines which part of the information based on the cell state is used for output, according to the current input tx t , the state of the hidden layer at the previous moment h t-1 And the latest cell state C t , through the interaction of sigmoid function and tanh function, the output h at the current moment is determined t ;
[0120] Sigmoid layer: o t =σ(W o ·[h t-1 ,x t ]+b o );
[0121] Output hidden state: h t =o t *tanh(C t );
[0122] Training process:
[0123] Forward propagation: For each time step of input, the hidden state and output are calculated according to the RNN;
[0124] Calculate loss: Use a loss function to measure the difference between the model prediction and the actual label;
[0125] Back propagation: Calculate the gradient of loss with respect to model parameters through the time back propagation algorithm;
[0126] Parameter update: Use Adam to update model parameters according to the gradient;
[0127] Testing and Reasoning:
[0128] The model performance is evaluated on the test set. During inference, the data to be classified is input into the trained RNN model. The model outputs the probability distribution of each category, and the category with the highest probability is taken as the classification result.
[0129] In one embodiment, for the above-mentioned data storage and management module, the data storage and management module includes a distributed file system, a cloud storage service unit, a data indexing and retrieval unit, a data backup unit and a fault recovery unit. The distributed file system is responsible for distributing and storing the video data on multiple nodes to achieve data redundancy and fault tolerance. Commonly used distributed file systems include GFS (Google File System), HDFS (Hadoop Distributed File System), etc. The cloud storage service unit uses the storage resources of cloud storage service providers (such as AWS, Azure, Alibaba Cloud, etc.) to store video data in the cloud. The data indexing and retrieval unit is responsible for establishing an index of the video data for fast retrieval and access. The index can be based on the metadata of the video (such as timestamp, location, event type, etc.) or the characteristics of the video content (such as images, sounds, etc.). The data backup unit is responsible for regularly backing up the video data to prevent data loss or damage. The backup can be stored locally or in the cloud, and can be configured as incremental backup or full backup. In the event of data loss, node failure or system crash, the fault recovery unit is responsible for recovering data from the backup. The fault recovery strategy may include data replication, data migration, data reconstruction, etc. The distributed file system will Part or all of the video data is stored in the cloud storage service unit to achieve remote backup and access to the data. The distributed file system and the cloud storage service unit transmit and synchronize data through an API interface or standard protocols (such as HTTP, HTTPS). The data index and retrieval unit accesses the video data in the distributed file system or cloud storage service, establishes and maintains indexes. The data backup unit periodically reads video data from the distributed file system or cloud storage service unit and backs it up to a designated storage location. The backup process involves operations such as data compression, encryption, and transmission. When a system failure occurs and data is lost, the fault recovery unit triggers the data recovery process. The fault recovery unit obtains the backup data from the data backup unit and then restores it to the original location or a new storage location.
[0130] In one embodiment, for the above-mentioned data encryption and security module, the data encryption and security module includes a data encryption unit, a secure communication unit, an access control and authority management unit, and a key management unit. The data encryption unit is responsible for encrypting the video data and using an encryption algorithm (such as AES, RSA, etc.) to convert the plaintext data into ciphertext data. The encryption process can be performed before the data is sent (transmission encryption) or when the data is stored (storage encryption). The secure communication unit is responsible for introducing end-to-end encryption technology and TLS / SSL protocol to ensure the security of data during transmission. End-to-end encryption means that the data is always encrypted when transmitted between the sender and the receiver, and the intermediate nodes cannot decrypt or view the data content. The TLS / SSL protocol The security of data transmission is ensured by encrypting the communication between the client and the server and verifying the identity of the server. The access control and permission management unit is responsible for establishing a complete access control and permission management mechanism to strictly control and review access to data. The access control and permission management unit will decide whether a user accesses specific video data based on the user's identity and permissions. Permission management may include role management, policy management, audit logs and other functions to ensure the compliance and traceability of data access. The key management unit is responsible for the generation, storage, distribution, update and destruction of encryption keys throughout their life cycle. The data encryption unit encrypts the video data and transmits it through the secure communication unit. The key management unit provides the data encryption unit with the keys required for encryption.
[0131] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. An intelligent video traffic monitoring system based on multi-viewpoints, It is characterized in that include: Video acquisition module: responsible for capturing video data in real time; Multi-viewpoint fusion processing module: responsible for spatial alignment and fusion of video images from multiple perspectives; Intelligent analysis module: responsible for building intelligent analysis models and conducting intelligent analysis; Data storage and management module: responsible for redundant storage and fault recovery of video data; Data encryption and security module: responsible for encrypting video data; User interaction and alarm module: responsible for providing the user interface and sending alarm information to relevant personnel when the system detects an abnormal situation; System configuration and management module: responsible for the overall configuration and management of the system; The video data captured in real time by the video acquisition module is transmitted to the multi-view fusion processing module through the Internet of Things protocol and communication standard. The multi-view fusion processing module transmits the fused video data to the intelligent analysis module. The intelligent analysis module transmits the analysis result to the data storage and management module for storage. At the same time, the intelligent analysis module requests the data storage and management module to provide historical data. When the intelligent analysis module detects an abnormal situation, the intelligent analysis module sends the alarm information to the user interaction and alarm module. The user interaction and alarm module requests to view the historical monitoring video or report. The data storage and management module is responsible for providing the historical monitoring video or report.
2. According to claim 1, a multi-viewpoint intelligent video traffic monitoring system, It is characterized in that The video acquisition module includes a camera unit, a data acquisition and processing unit, a data transmission unit and a control and management unit. The camera unit is used to capture video data of traffic scenes. The data acquisition and processing unit is responsible for receiving raw video data from the camera and performing preliminary processing. The data transmission unit uses the Internet of Things protocol and communication standards to transmit the processed video data to the multi-view fusion processing module in real time. The control and management unit is responsible for remote control and management of the camera. At the same time, the control and management unit is responsible for monitoring the operating status of the camera. The camera unit sends the captured video data of the traffic scene to the data acquisition and processing unit. The data acquisition and processing unit sends the processed video data to the data transmission unit. The control and management unit sends control instructions to the camera unit.
3. According to claim 1, a multi-viewpoint intelligent video traffic monitoring system, It is characterized in that The multi-viewpoint fusion processing module includes a video receiving unit, an image preprocessing unit, an image registration unit, an image fusion unit, an occlusion correction and prediction unit and a result output unit. The video receiving unit is responsible for receiving video data streams from cameras at different angles, the image preprocessing unit is responsible for preprocessing the received video frames, the image registration unit is responsible for spatially aligning the video data streams from cameras at different angles using an image registration algorithm, the image fusion unit is responsible for fusing the video data from cameras at different angles based on image registration, the occlusion correction and prediction unit is responsible for combining deep learning technology to correct and predict occlusion phenomena, and by training a deep learning model to learn the characteristics and motion laws of occlusion objects, The position and state of the occluded object can be inferred when it is generated. The result output unit is responsible for outputting the fused video image and the results of occlusion correction and prediction to the subsequent intelligent analysis module or user interaction module. The video receiving unit sends the video data streams received from cameras of different angles to the image preprocessing unit for preliminary processing. The image preprocessing unit sends the preprocessed video frames to the image registration unit for spatial alignment. The image registration unit sends the registered video images to the image fusion unit for fusion processing. During the fusion processing, the occlusion correction and prediction units work simultaneously to correct and predict the occlusion phenomenon in the fused image. The result output unit receives the video image after the fusion processing is completed and the occlusion phenomenon is corrected and predicted.
4. According to claim 3, a multi-viewpoint intelligent video traffic monitoring system, It is characterized in that The occlusion correction and prediction unit uses Transformer to correct and predict the occlusion phenomenon, including the following steps: Data preprocessing: obtain images or video sequences containing occlusion phenomena, and perform data cleaning on the images or video sequences containing occlusion phenomena; Feature extraction: Transformer input: convert the preprocessed image or video sequence into an input format acceptable to the Transformer model; Position coding: Add position coding. The specific formula is as follows: ON (pos,2i) =sin(pos / 1000 2i / dmodel ); ON (pos,2i+1) =cos(pos / 1000 2i / dmodel ); Among them, PE (pos,i) Represents the encoded value at position pos, where pos is the position index of the current element in the sequence, i is the dimension index, and is used to traverse each dimension of the position encoding vector. model is the dimension of the Transformer model, i.e. the dimension of the input embedding; Self-attention mechanism: The Transformer model extracts global features from an image or video sequence through a self-attention mechanism, which allows the model to take into account other elements in the sequence when processing each element, thereby capturing the dependencies between elements; Occlusion detection and recognition: Feature analysis: Analyze the extracted features and identify occlusions in images or video sequences; Occlusion modeling: Model the detected occlusions and analyze their properties; Occlusion correction: Local feature fusion: Use the Transformer model to fuse local features before and after occlusion to restore the information of the occluded part; Global feature alignment: Use global features to guide the correction of occluded parts; predict: Time series prediction: For video sequences, the sequence modeling capability of the Transformer model is used to predict the target state during occlusion. Trajectory prediction: Based on the target trajectory and feature changes before occlusion, the target's motion trajectory and position during occlusion are predicted; Loss function optimization: Optimize the model parameters by defining a suitable loss function; Output results: Output target prediction results during occlusion.
5. The multi-viewpoint intelligent video traffic monitoring system according to claim 1, It is characterized in that The intelligent analysis module includes a data input unit, a preprocessing unit, a model training unit and an intelligent analysis unit. The data input unit is responsible for receiving the fused video data and historical data provided by the multi-viewpoint fusion processing module and the data storage and management module as input data. The preprocessing unit preprocesses the input data. The model training unit uses LSTM and large-scale data sets for training. The intelligent analysis unit combines traffic flow theory, road network structure and weather conditions multi-source information to build an intelligent analysis model and realize the intelligent analysis function. The data input unit sends the received input data to the preprocessing unit for cleaning and formatting. The preprocessing unit uses the preprocessed data as training data for the deep learning model, and the trained deep learning model is deployed in the intelligent analysis unit.
6. The multi-viewpoint intelligent video traffic monitoring system according to claim 5, It is characterized in that The model training unit uses LSTM and a large-scale data set for training, including the following steps: Data preparation: Collect the fused video data and historical data provided by the multi-view fusion processing module and the data storage and management module, extract the feature data related to the target task, and divide the data set into a training set, a validation set, and a test set, with a ratio of 70%, 15%, and 15%; Model construction: Building a model using LSTM: Forget gate: f t =σ(W f ·[h t-1 ,x t ]+b f ) Among them, W f is the weight matrix of the forget gate, b f is the bias term, σ is the sigmoid function, [h t-1 ,x t ] means to set h t-1 and x t Concatenate into a vector; Input gate: The input gate is responsible for updating the cell state. The input gate consists of two parts: one is the sigmoid layer, which determines which information will be updated; the other is the tanh layer, which creates a new candidate value vector. The new candidate value vector is added to the cell state. Finally, the two information are multiplied to update the cell state. Sigmoid layer: i t = σ(W i · [h t-1 , x t + b i ); Tanh layer: Cell status update: Output gate: The output gate determines which part of the information based on the cell state is used for output, according to the current input tx t , the state of the hidden layer at the previous moment h t-1 And the latest cell state C t , through the interaction of sigmoid function and tanh function, the output h at the current moment is determined t ; Sigmoid layer: o t = σ(W o · [h t-1 , x t + b o ); Output hidden state: h t =o t *tanh(C t ); Training process: Forward propagation: For each time step of input, the hidden state and output are calculated according to the RNN; Calculate loss: Use a loss function to measure the difference between the model prediction and the actual label; Back propagation: Calculate the gradient of loss with respect to model parameters through the time back propagation algorithm; Parameter update: Use Adam to update model parameters according to the gradient; Testing and Reasoning: The model performance is evaluated on the test set. During inference, the data to be classified is input into the trained model. The model outputs the probability distribution of each category, and the category with the highest probability is taken as the classification result.
7. The multi-viewpoint intelligent video traffic monitoring system according to claim 1, It is characterized in that The data storage and management module includes a distributed file system, a cloud storage service unit, a data index and retrieval unit, a data backup unit and a fault recovery unit. The distributed file system is responsible for distributing and storing video data on multiple nodes. The cloud storage service unit uses the storage resources of the cloud storage service provider to store video data in the cloud. The data index and retrieval unit is responsible for establishing an index of video data. The data backup unit is responsible for regularly backing up video data. When data is lost, a node fails or the system crashes, the fault recovery unit is responsible for recovering data from the backup. The distributed file system stores part or all of the video data in the cloud storage service unit to achieve remote backup and access of data. The distributed file system and the cloud storage service unit perform data transmission and synchronization through an API interface or a standard protocol. The data index and retrieval unit accesses the video data in the distributed file system or cloud storage service, establishes and maintains an index. The data backup unit regularly reads video data from the distributed file system or cloud storage service unit. When a system failure occurs and data is lost, the fault recovery unit triggers a data recovery process. The fault recovery unit obtains the backup data from the data backup unit and then restores it to the original location or a new storage location.
8. The multi-viewpoint intelligent video traffic monitoring system according to claim 1, It is characterized in that The data encryption and security module includes a data encryption unit, a secure communication unit, an access control and permission management unit, and a key management unit. The data encryption unit is responsible for encrypting the video data. The secure communication unit is responsible for introducing end-to-end encryption technology and TLS / SSL protocol. The access control and permission management unit is responsible for establishing a complete access control and permission management mechanism to strictly control and review access to data. The access control and permission management unit will decide whether a user accesses specific video data based on the user's identity and permissions. The key management unit is responsible for the full life cycle management of encryption keys. After the data encryption unit encrypts the video data, it is transmitted through the secure communication unit. The key management unit provides the data encryption unit with the key required for encryption.
Citation Information
Cited By
Traffic accident ranging system based on computer vision
CN120445163A
Multi-camera cooperative monitoring method and system, electronic equipment and medium
CN120897037A
