Illegal snapshot method and system based on image recognition

Abstract

The invention discloses a violation snapshot method based on image recognition, which relates to the related technical field of traffic video monitoring and comprises the steps of environment perception, scene analysis, scene feature library establishment, multi-modal data acquisition, server model aggregation, model issuing and fine adjustment and violation behavior recognition. The invention further discloses a violation snapshot system based on image recognition. The violation snapshot system comprises an environment sensing module, a data acquisition module, a data processing center, a violation behavior recognition module and a snapshot and recording module. Different from a fixed snapshot strategy of a traditional method, the system can dynamically adjust the snapshot strategy according to real-time understanding of a traffic scene, the system can automatically improve the snapshot frame rate when detecting a high-risk scene in which a traffic accident is about to occur, and the snapshot frame rate can be automatically increased in a road section in which the traffic flow is small and the scene is simple. Snapshot resource consumption can be properly reduced; therefore, a complex and changeable traffic environment can be better dealt with, and the snapshot effectiveness and the resource utilization efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field related to traffic video monitoring, and in particular to a method and system for capturing violations based on image recognition. Background Art

[0002] In the past, the supervision of traffic violations mainly relied on on-site enforcement by traffic police. This method has many disadvantages. For example, during peak hours or on wide roads, it is difficult for traffic police to cover all areas, resulting in many violations being difficult to detect and deal with in a timely manner. In addition, manual enforcement is easily affected by human factors such as fatigue and observation angle, making accuracy and consistency difficult to guarantee.

[0003] Early image recognition-based violation capture mainly relied on simple image comparison and rule judgment; this method frequently encountered problems in complex environments. For example, in insufficient light, bad weather (such as heavy rain and fog), and vehicle obstruction, the image quality degraded and the recognition accuracy was greatly reduced. In addition, for some complex violations, such as illegal lane changes under special road conditions (controversial lane changes due to avoiding pedestrians or other emergencies), it is difficult to accurately determine whether it is a violation based on images alone.

[0004] The process of collecting and processing traffic data involves a large amount of sensitive information of vehicles and car owners; traditional data collection and transmission methods have the risk of data leakage and may infringe on the privacy of car owners.

[0005] Building an accurate violation identification model requires a large amount of data for training. Traffic scenarios in different regions vary greatly, and the trained model may face the problem of poor adaptability when applied to new environments.

[0006] Therefore, it is necessary to propose a violation capture method and system based on image recognition to solve the above problems. Summary of the Invention

[0007] In response to the shortcomings of the existing technology, the present invention provides a method and system for capturing violations based on image recognition, which solves the problems that early image recognition-based violation capture was difficult to accurately determine whether it was a violation, the data collection and transmission methods had the risk of data leakage, and the model may face poor adaptability when applied to new environments.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A method for capturing violations based on image recognition, comprising the following steps:

[0010] Environmental perception: Deploy multiple types of sensors to collect visual, vehicle speed and distance, and meteorological data; use sensor fusion methods to integrate the data to obtain environmental perception information;

[0011] Scene analysis: A deep learning-based scene understanding algorithm is used to analyze perceived traffic scenes. The algorithm identifies road types, traffic signs and markings, road conditions, and dynamic changes in the surrounding environment to extract scene features.

[0012] Establish a scene feature library: classify and label traffic scenes according to different scene features, and the system automatically adjusts the capture strategy;

[0013] Multimodal data acquisition: Multimodal data is collected at several data collection nodes through cameras, microphones, millimeter-wave radars, vehicle networking interfaces, and communication receiving equipment. The collected data is processed and feature extracted, the extracted features are encrypted, and the processed data is fused.

[0014] Server-side model aggregation: The federated learning server receives encrypted feature data from multiple clients and uses a federated learning algorithm to perform a weighted average of the local models based on the weights of the multiple client data to obtain a global model.

[0015] Model distribution and fine-tuning: The updated global model is distributed to multiple clients, which then fine-tune the global model based on local data to adapt to local traffic scenarios.

[0016] Violation identification: Based on the updated federated learning model, comprehensive analysis of multimodal data is performed to identify vehicle violations.

[0017] Optionally, the steps for implementing the scenario analysis are as follows:

[0018] CNN model design: Design a convolutional neural network (CNN) model for traffic scene understanding, including multiple convolutional layers, pooling layers, and fully connected layers;

[0019] Training data preparation: Collect a number of annotated traffic scene image data, including images under different weather, lighting, and road conditions;

[0020] Model training: Use labeled data to train the CNN model and optimize model parameters to minimize classification error;

[0021] Real-time analysis: Deploy the trained CNN model to the actual system to extract features and classify the real-time collected image data;

[0022] A convolutional neural network (CNN) is used to analyze perceived traffic scenes in real time; CNN automatically extracts features from images and performs classification and recognition.

[0023] Optionally, the steps for establishing the scene feature library are as follows:

[0024] Feature extraction: extracting feature vectors from the middle layers of the CNN model;

[0025] Feature library construction: Classify and label traffic scenes according to different scene characteristics to build a scene feature library; use the support vector machine classification algorithm to classify scenes and determine the risk level of the current scene.

[0026] Optionally, the steps for comprehensive analysis of multimodal data in violation identification are as follows:

[0027] The model considers vehicle motion in images, anomalies in acoustic signals, driving parameter anomalies in onboard sensor data, and potential violations in communication signals;

[0028] Image data P image : Use convolutional neural networks to identify vehicle actions in images;

[0029] Sound signal P autio : Apply audio processing algorithms to detect anomalies in sound signals;

[0030] Vehicle sensor data P sensor :Use time series analysis method to detect driving parameter anomalies in vehicle sensor data;

[0031] Communication signal P comm : Collect communication data between vehicles and between vehicles and infrastructure through communication receiving equipment to analyze potential violation information;

[0032] Federated learning model: The recognition results of multimodal data are used as input and fed into the federated learning model for comprehensive analysis. The mathematical representation is as follows:

[0033] P federated =Federated Learning Model(P image ,P audio ,P sensor ,P comm ),

[0034] Among them, P federated It is the output of the federated learning model, indicating the identification result of the violation.

[0035] Optionally, the violation capture method further includes a judgment rule library, which sets corresponding judgment thresholds and conditions according to different violation types to determine whether a violation occurs.

[0036] Optionally, the violation capture method further includes capture control and recording, in the following manner: when a violation is identified, a capture mechanism is triggered, and the capture system adjusts camera parameters, including vehicle front, side, and license plate number information, and records the time, location, multimodal data features, and violation type of the violation, and uploads the information to a database, where the records are encrypted and stored using blockchain technology to prevent data tampering and leakage;

[0037] The camera parameter adjustment method in the capture mechanism is as follows: when a violation is identified, the capture mechanism is triggered and the camera parameters are adjusted to ensure that a clear and complete image of the violating vehicle is captured;

[0038] The method of recording violations is to record the time, location, multimodal data characteristics, and violation type information of the violation.

[0039] A violation capture system based on image recognition, comprising:

[0040] Environmental perception module: including camera module, millimeter-wave radar module and meteorological sensor module;

[0041] Data acquisition module: obtains raw data from sensors and transmits it to the data processing center for preprocessing and feature extraction;

[0042] Data processing center: Processes camera, millimeter-wave radar, and meteorological data; extracts visual features, vehicle information, and environmental features; uses convolutional neural networks to analyze the fused data in real time and extract scene features;

[0043] Multimodal data acquisition module: collects multimodal data at each data acquisition node through cameras, microphones, millimeter-wave radars, vehicle networking interfaces, and communication receiving devices;

[0044] Violation identification module: Based on the updated federated learning model, it conducts comprehensive analysis of multimodal data to identify vehicle violations.

[0045] The beneficial effects of the present invention are as follows:

[0046] 1. Unlike the fixed capture strategy of traditional methods, the present invention's system can dynamically adjust the capture strategy based on real-time understanding of the traffic scene. When a high-risk scene of an impending traffic accident is detected, the system will automatically increase the capture frame rate, focusing on vehicles that may be involved in the accident. On roads with less traffic and simple scenes, the capture resource consumption can be appropriately reduced. This adaptive capability enables the system to better cope with complex and changing traffic environments, improving the effectiveness of capture and resource utilization efficiency.

[0047] 2. The present invention uses environmental information obtained from dynamic scene understanding to assist in the determination of violations. In severe weather conditions such as snowy or foggy days, the system will adjust the judgment criteria for the vehicle's driving trajectory according to weather characteristics, taking into account factors such as reduced visibility and changes in road adhesion, to more accurately determine whether the vehicle has violated regulations, such as whether a safe distance is maintained or whether the vehicle is speeding. At the same time, for different road types, corresponding violation determination rules are formulated according to road characteristics, making violation capture more accurate and reasonable.

[0048] 3. In the process of multimodal data collection, the present invention realizes data privacy protection and secure sharing through a federated learning framework; each data collection node only needs to upload encrypted feature data without sharing the original data, thus avoiding the risk of data being stolen or leaked during transmission.

[0049] 4. The multimodal federated learning of the present invention allows model training on multiple distributed nodes. Each node can train and optimize the local model according to the characteristics of the traffic scenario in which it is located; then, through the model aggregation operation on the server side, the local models of each node are integrated into a global model, and then the global model is distributed to each node for personalized fine-tuning; this method can not only use global data to learn common violation behavior characteristics, but also personalize the model according to the traffic characteristics of different regions and different road sections, so that the violation capture model can perform well in various scenarios.

[0050] 5. In addition to traditional image data, the present invention also integrates data from multiple modalities such as sound, on-board sensors, and communication signals. Sound signals can provide additional clues for determining violations. For example, by analyzing the braking sound, it can be determined whether the vehicle is emergency braking and the timeliness of braking. On-board sensor data can directly reflect the driving status of the vehicle and plays a key role in identifying violations such as speeding and improper acceleration. Communication signal data can reveal the interaction between vehicles and potential violations, such as the situation where a vehicle fails to send communication signals as required at an intersection, causing traffic congestion. The comprehensive use of multimodal data greatly enriches the information source for violation identification and reduces misjudgments and missed judgments caused by the limitations of single-modal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The figure is a flow chart of the method for capturing violations based on image recognition according to the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] Example 1

[0054] See also Figure 1 , a method for capturing illegal activities based on image recognition, comprising the following steps:

[0055] Step 1: Environmental Perception: Deploy multiple types of sensors, including cameras, millimeter-wave radars, and meteorological sensors, to collect visual information, vehicle speed and distance, and meteorological data. Cameras capture visual information about vehicles and roads, including vehicle shape, color, license plates, traffic signs and markings, pedestrians, and non-motorized vehicles. Millimeter-wave radars monitor vehicle speed, distance, and angle information. Meteorological sensors collect weather conditions, including light intensity, rainfall, and fog concentration. Using sensor fusion methods, data from different sensors is integrated to obtain comprehensive environmental perception information.

[0056] Step 2: Scene Analysis: A deep learning-based scene understanding algorithm is used to analyze the perceived traffic scene. The algorithm identifies road types, traffic signs and markings, road conditions, and dynamic changes in the surrounding environment to extract scene features.

[0057] Step 3: Establish a scene feature library: Classify and label traffic scenes according to different scene features, and the system automatically adjusts the capture strategy;

[0058] Step 4: Multimodal data acquisition: Multimodal data is collected at several data acquisition nodes through cameras, microphones, millimeter-wave radars, vehicle networking interfaces, and communication receiving devices. The collected data is processed and feature extracted, the extracted features are encrypted, and the processed data is fused.

[0059] Step 5: Server-side model aggregation: The federated learning server receives encrypted feature data from multiple clients and uses a federated learning algorithm to perform a weighted average of the local models based on the weights of the data from multiple clients to obtain a global model.

[0060] Step 6: Model distribution and fine-tuning: The updated global model is distributed to multiple clients. The clients fine-tune the global model based on local data to adapt to the characteristics of local traffic scenarios.

[0061] Step 7: Violation Identification: Based on the updated federated learning model, comprehensive analysis of multimodal data is performed to identify vehicle violations.

[0062] The implementation steps of the sensor fusion method in environmental perception are as follows:

[0063] Data acquisition: Obtaining raw data from cameras, millimeter-wave radars, and weather sensors;

[0064] Data preprocessing: De-noise and enhance contrast on camera data; filter noise on millimeter-wave radar data and convert it from polar coordinates to Cartesian coordinates; and normalize meteorological data.

[0065] Feature extraction: extract visual features (such as edges and textures) from camera data, extract vehicle information (such as speed, distance, and angle information) from millimeter-wave radar data; extract environmental features (such as light intensity and rainfall) from meteorological data;

[0066] Sensor fusion: Kalman filtering is used to fuse visual features and environmental features to obtain comprehensive environmental perception information. The goal of sensor fusion is to integrate data obtained by different sensors to build a comprehensive traffic scene perception model. The mathematical representation of the fusion method is:

[0067] The state prediction equation is: in, is the predicted state vector, A is the state transfer matrix, is the state estimate of the previous moment, B is the control input matrix, u k is the control input;

[0068] The state update equation is: in, is the updated state vector, K k is the Kalman gain, z k is the observation vector at the current moment, and H is the observation matrix.

[0069] In this embodiment, the accuracy of violation detection is greatly improved by integrating data from multiple sensor types, including high-resolution cameras, millimeter-wave radar, and meteorological sensors. For example, in complex weather conditions, such as heavy fog or heavy rain, meteorological sensors can provide timely feedback on weather conditions. When camera vision is obscured by inclement weather, radar data can assist in determining the vehicle's location and speed.

[0070] For some partially obscured illegal vehicles, such as a small car changing lanes illegally with part of its body obscured by a large truck, the camera may not be able to obtain complete vehicle information, but the radar can provide the vehicle's distance and speed change information. Combined with some features captured by the camera, sensor fusion technology can more accurately determine whether the vehicle has violated the rules.

[0071] Example 2

[0072] This embodiment further optimizes the first embodiment. Specifically, scene analysis uses a deep learning-based scene understanding algorithm to perform real-time analysis of perceived traffic scenes. The algorithm identifies road types (e.g., urban roads, highways, rural roads), traffic sign and road marking status (clearness and integrity), road conditions (e.g., congested, unobstructed, slippery), and dynamic changes in the surrounding environment (e.g., the sudden appearance or movement of pedestrians or non-motorized vehicles) to extract scene features.

[0073] The implementation steps of scenario analysis are as follows:

[0074] CNN model design: Design a convolutional neural network (CNN) model for traffic scene understanding, including multiple convolutional layers, pooling layers, and fully connected layers;

[0075] Training data preparation: Collect a number of annotated traffic scene image data, including images under different weather, lighting, and road conditions;

[0076] Model training: Use labeled data to train the CNN model and optimize model parameters to minimize classification error;

[0077] Real-time analysis: Deploy the trained CNN model to the actual system to extract features and classify the real-time collected image data;

[0078] A convolutional neural network (CNN) is used to analyze the perceived traffic scene in real time. CNN can automatically extract features from images and perform classification and recognition. The mathematical formula for the convolution operation is:

[0079] (I*K)(x,y)=∑ i,j I(x+i,y+j)·K(i,j),

[0080] Where I is the input image, K is the convolution kernel, and (x, y) is the coordinate of the output image;

[0081] The activation function is ReLU (Rectified Linear Unit): f(x) = max(0,x);

[0082] Establish a scene feature library: Traffic scenes are classified and labeled based on different scene characteristics. For example, a scene with a slippery road in rainy weather is labeled as a "high-risk scene." In this case, the judgment criteria for vehicle violations need to be more stringent, and the system will automatically adjust the capture strategy to improve accuracy.

[0083] The steps to build a scene feature library are as follows:

[0084] Feature extraction: Extract feature vectors from the middle layer of the CNN model, f = [f1, f2, ..., f n ] T , where f i is the i-th feature, n is the total number of features;

[0085] Feature library construction: Classify and label traffic scenes according to different scene features (such as weather, lighting, and road conditions) to build a scene feature library; use the support vector machine (SVM) classification algorithm to classify the scenes and determine the risk level of the current scene. The mathematical formula is: y = sign (w T f+b), where w is the weight vector, b is the bias term, and y is the classification result.

[0086] In this embodiment, the deep learning-based scene understanding algorithm can analyze traffic scenes in real time and accurately identify various violations. Whether it is common violations such as running red lights, speeding, illegal lane changes, or more complex violations such as cutting in line on congested roads or illegal turns at intersections where turns are not allowed, the algorithm can make accurate judgments through scene understanding and analysis. For different types of roads (urban roads, highways, rural roads), the algorithm can perform targeted violation identification based on road characteristics and traffic regulations. For example, on highways, it can accurately identify whether a vehicle is traveling below the minimum speed limit (in some congestion relief areas or specific construction sections, there are minimum speed limits), and whether there are violations such as improper lane occupation (such as occupying the emergency lane for a long time).

[0087] The established scene feature library can classify and label traffic scenes according to different scene characteristics, allowing the violation capture system to flexibly adapt to various complex traffic scenarios. In "high-risk scenarios" such as rainy days and slippery roads, the system automatically adjusts its capture strategy and adopts stricter criteria for determining vehicle violations. For example, under normal road conditions, a minor violation of the line may only be a warning, but in high-risk scenarios such as slippery roads, crossing the line may lead to more serious consequences such as loss of vehicle control, and the system will capture it as a violation. Moreover, when marked as a "high-risk scenario", the system can adjust the capture sensitivity to capture potentially dangerous violations more promptly, such as a vehicle braking too short a distance on a slippery road.

[0088] Scene analysis can perceive the dynamic changes of the surrounding environment in real time, including the sudden appearance or movement of pedestrians and non-motor vehicles; when capturing violations, the system can fully consider these dynamic factors to avoid misjudgment; for example, when a bicycle suddenly drives from the roadside into the motor vehicle lane, the vehicle behind takes emergency evasive action to avoid collision. The system can judge that this is not a violation of the vehicle, but a reasonable risk avoidance measure based on its dynamic understanding of the scene; at the same time, real-time analysis of road conditions (such as congestion, smooth flow, wetness, etc.) also helps to accurately capture violations; in congested sections, the system can focus on violations such as vehicles cutting in line and changing lanes at will; in unobstructed sections, it pays more attention to monitoring violations such as speeding.

[0089] Example 3

[0090] This embodiment is a further optimization based on Example 2. Specifically, multimodal data collection: multimodal data is collected at each data collection node (such as capture equipment at different traffic intersections and road sections) through cameras, microphones, millimeter-wave radars, vehicle networking interfaces, and communication receiving equipment, including vehicle image data, sound signals (such as brake sounds and horn sounds), vehicle-mounted sensor data (such as vehicle speed, acceleration, and steering angle, obtained through vehicle-to-vehicle technology), and communication signal data (such as vehicle-to-vehicle and vehicle-to-infrastructure V2X communication data); the collected multimodal data is locally processed and feature extracted (for example, running a convolutional neural network (CNN) on the image data to extract image features, applying an audio processing algorithm to the sound signal to extract sound features, and using a time series analysis method to extract data features on the vehicle-mounted sensor data); the extracted features are encrypted (homomorphically encrypting the extracted features and adding differential privacy noise during the feature extraction process to protect data privacy and ensure data privacy and security); the pre-processed multimodal data is fused to provide a reliable foundation for subsequent violation identification and capture;

[0091] Image data preprocessing: Gaussian filtering is used to remove image noise, histogram equalization is used to enhance image contrast, and a perspective transformation matrix is ​​used to correct image perspective distortion to improve image quality. For image data, preprocessing operations such as denoising, contrast enhancement, and perspective distortion correction improve image quality and reduce recognition errors caused by image quality issues. Even in low-light environments or inclement weather, preprocessed images can still clearly present vehicle features, ensuring the stable operation of the subsequent image recognition-based violation judgment process, and preventing frequent errors caused by image quality fluctuations.

[0092] Sound signal preprocessing: A bandpass filter is used to remove ambient noise interference and extract key sound signal features, such as frequency and amplitude. After filtering the sound signal to remove ambient noise interference, the extracted key sound features (such as frequency and amplitude) are more representative and stable, and can accurately reflect vehicle sound-related violations. For example, the frequency and amplitude of the horn can be used to determine whether the horn is honking illegally, without misjudging it due to interference from noisy ambient sounds.

[0093] Preprocessing of on-board sensor data and communication signal data: This involves converting and calibrating speed data transmitted from different vehicles to a unified unit, normalizing the data to a uniform range, and ensuring data consistency and accuracy. For example, speed data transmitted from different vehicles is converted and calibrated to a unified unit to eliminate the impact of sensor errors.

[0094] Calibration and normalization of vehicle sensor data and communication signal data ensures that data from sensors of different vehicles and brands can be compared and analyzed under unified standards. Taking vehicle speed data as an example, speed sensors of different vehicles may have certain errors. Through calibration and normalization, various vehicle speed data are unified into standard units and accuracy ranges, providing the system with a reliable basis for judging violations such as speeding, avoiding misjudgments or missed judgments due to data discrepancies.

[0095] Server-side model aggregation: The federated learning server receives encrypted feature data from each client and uses a federated learning algorithm, such as an improved version of the federated averaging algorithm, to perform a weighted average of the local models based on the weights of each client's data (determined by data quality and volume) to obtain a global model.

[0096] The federated averaging algorithm in server-side model aggregation is: Among them, W global is the global model parameter, W i is the local model parameter of the i-th client, w i is the weight of the i-th client, N is the number of clients;

[0097] The weight is determined based on data quality and data volume factors, and the mathematical model is: Among them, Data Quality i is the data quality of the i-th client;

[0098] Model distribution and fine-tuning: The updated global model is distributed to each client, and the client fine-tunes the global model based on local data to adapt to the characteristics of the local traffic scene. For example, at an intersection where special traffic conditions often occur, the client adjusts the global model based on local data to more accurately identify violations at that intersection.

[0099] In this embodiment, feature extraction is performed on various modal data such as images, sound signals, and vehicle-mounted sensor data on each client. For example, by extracting image features through a convolutional neural network (CNN), key visual information such as the vehicle's appearance, color, and license plate can be accurately captured. Audio processing algorithms can extract sound features, which helps to identify special sound events such as vehicle horns and brakes. Time series analysis methods are used for vehicle-mounted sensor data to effectively extract dynamic change features such as vehicle speed and acceleration. Fusion of these different modal features in a federated learning framework can provide the model with more comprehensive and rich information, thereby more accurately judging violations. For example, when judging whether a vehicle has run a red light, it is possible not only to determine whether the vehicle has crossed the stop line in the image, but also to determine whether there is an emergency brake sound based on the sound signal, and to analyze the vehicle's driving state when approaching the intersection through vehicle-mounted sensor data, and to make a more accurate judgment based on multiple features.

[0100] Using an advanced federated learning algorithm, local models are weighted averaged based on the weight of each client's data (which can be determined based on factors such as data quality and volume) to form a global model. This algorithm fully leverages the data strengths of each client, giving higher weight to clients with high-quality and large-volume data, making the aggregated global model more representative and accurate. For example, at busy intersections with high traffic volume and diverse violations, the collected data is high-quality and abundant, accounting for a larger weight in the model aggregation process. This allows the complex traffic scene characteristics of the intersection to be incorporated into the global model, improving the global model's ability to identify various violations.

[0101] The updated global model is regularly distributed to each client, which then fine-tunes the model based on local data to adapt to the specific characteristics of the local traffic scenario. This mechanism not only ensures the overall performance of the model, but also enables personalized adjustments for the unique traffic conditions at different intersections or road sections. For example, at an intersection prone to unusual traffic conditions (such as the mixed flow of people and vehicles at a school during rush hour), the client can adjust the global model based on local data to more accurately identify violations at that intersection, such as vehicles temporarily stopping in prohibited areas to pick up or drop off students. This improves the applicability and accuracy of the violation capture system in specific scenarios.

[0102] Traditionally, image recognition alone can be used to identify violations, potentially leading to misjudgments due to factors such as occlusion, insufficient lighting, and image blur. For example, at an intersection, a vehicle may appear to have run a red light due to being temporarily obscured by a larger vehicle ahead. However, combined with data from onboard sensors, such as the vehicle's speed, acceleration, and steering angle, it can be used to determine whether the vehicle actually ran a red light. If the vehicle maintained a normal speed before being obscured and showed no unusual behavior such as acceleration or steering, it likely did not run a red light, thus avoiding misjudgments. Sound signals can also provide crucial evidence. For example, when determining whether a vehicle has honked its horn, an image may only show its location, while the sound signal collected by a microphone array can clearly indicate whether a horn was honking and further identify the vehicle by the direction and intensity of the sound source, enabling accurate judgment. Communication signal data also plays a significant role. In a connected vehicle environment, vehicles exchange information in real time with each other and with infrastructure. For example, if a vehicle ahead suddenly brakes and broadcasts its braking signal over the network, and a following vehicle follows suit but is still misjudged by image recognition as an inappropriate braking violation, communication signal data can reveal the true situation and correct the misjudgment.

[0103] In complex traffic scenarios, such as road construction and traffic accident scenes, multimodal data can enable the violation capture system to better understand the scene; image data shows visual information such as construction area signs and vehicle positions, on-board sensor data reflects changes in the vehicle's driving status near the area, sound signals may contain environmental characteristics such as construction machinery noise, and communication signals may contain traffic diversion instructions issued by relevant departments; by integrating these multimodal data, the system can accurately determine whether the vehicle's behavior in complex scenarios is illegal; for example, in cases where road construction causes the lane to narrow and there are temporary traffic signs, the vehicle appears to be driving over the line, but combined with the speed and steering angle displayed by the on-board sensors and the traffic guidance information received by communication, it can be determined that the vehicle is reasonably avoiding rather than illegally crossing the line.

[0104] Example 4

[0105] This embodiment is a further optimization based on Example 3. Specifically, violation identification: Based on the updated federated learning model, multimodal data is comprehensively analyzed to identify automobile violations. The steps for comprehensively analyzing multimodal data in violation identification are as follows:

[0106] The model considers vehicle actions in images (such as running a red light or changing lanes across a solid line), anomalies in sound signals (such as illegal honking), driving parameter anomalies in on-board sensor data (such as speeding or sudden braking that do not conform to normal driving patterns), and potential violations in communication signals (such as dangerous driving behaviors caused by inappropriate communication interactions between vehicles).

[0107] Image data: Use convolutional neural network (CNN) to identify the vehicle action in the image, mathematically expressed as: P image =CNN(I), where I is the input image, P image is the recognition result of the vehicle action in the image;

[0108] Sound signal: Apply audio processing algorithms (such as Mel-frequency cepstral coefficients MFCC) to detect anomalies in the sound signal, which is mathematically expressed as: P audio =MFCC(A), where A is the input sound signal, P audio is the detection result of anomalies in the sound signal;

[0109] Vehicle sensor data: Use time series analysis methods (such as LSTM) to detect driving parameter anomalies in vehicle sensor data, which can be expressed mathematically as: P sensor =LSTM(S), where S is the input vehicle sensor data, P sensor It is the detection result of abnormal driving parameters;

[0110] Communication signal: The communication data between vehicles and between vehicles and infrastructure is collected through communication receiving equipment to analyze potential violation information. The mathematical expression is: P comm =Analyze(C), where C is the input communication signal, P comm It is the detection result of potential violation information;

[0111] Federated learning model: The recognition results of multimodal data are used as input and fed into the federated learning model for comprehensive analysis. The mathematical representation is as follows:

[0112] P federated =Federated Learning Model(P image ,P audio ,P sensor ,P comm ),

[0113] Among them, P federated It is the output of the federated learning model, indicating the identification result of the violation.

[0114] In this embodiment, based on the updated federated learning model, a comprehensive analysis of multimodal data is performed, which can comprehensively and accurately identify automobile violations; by integrating vehicle movements in images, anomalies in sound signals, driving parameter anomalies in on-board sensor data, and potential violation information in communication signals, misjudgments or missed judgments that may exist in a single data source are avoided; for example, when judging whether a vehicle has run a red light, it is not only based on whether the vehicle has crossed the stop line in the image, but also on whether the vehicle has received the traffic light signal at the intersection in the communication signal and the driving status of the vehicle when approaching the intersection in the on-board sensor data (such as whether it has slowed down), so as to more accurately judge violations and greatly improve the accuracy of violation identification.

[0115] Example 5

[0116] This embodiment is further optimized on the basis of embodiment 4. Specifically, the violation capture method also includes a judgment rule library. The judgment rule library sets corresponding judgment thresholds and conditions according to different violation types to determine whether a violation has occurred. For example, for speeding violations, the judgment is made based on the road speed limit signs and the vehicle speed data transmitted by the on-board sensors; for illegal honking, the judgment is made based on the intensity and frequency of the sound signal and the no-honking regulations in the area.

[0117] In this embodiment, by establishing a rule base for determining violations, corresponding judgment thresholds and conditions are set according to different violation types, so that the judgment of violations has a clear and scientific basis; for speeding violations, the road speed limit signs and the vehicle speed data transmitted by the on-board sensors are combined for judgment, which can adapt to the speed limit requirements of different roads and accurately identify speeding vehicles; in no-honking areas, the judgment of illegal honking is based on the intensity and frequency of the sound signal and the no-honking regulations of the area, avoiding misjudgment due to environmental noise or other similar sounds; this rule-based judgment method ensures that there are rules to follow for the identification of various violations, thereby improving the reliability and fairness of the entire system.

[0118] Example 6

[0119] This embodiment is a further optimization based on Example 5. Specifically, the violation capture method also includes capture control and recording. The method is as follows: when a violation is identified, the capture mechanism is triggered, and the capture system adjusts camera parameters (such as focal length and shutter speed) to ensure that a clear and complete image of the violating vehicle is captured, including the front and side of the vehicle and the license plate number information. The time, location, multimodal data characteristics (such as images, sounds, sensor data), and violation type of the violation are recorded, and the information is uploaded to the database. The records are encrypted and stored using blockchain technology to prevent data tampering and leakage.

[0120] The camera parameter adjustment method in the capture mechanism is as follows: when a violation is identified, the capture mechanism is triggered and the camera parameters (such as focal length and shutter speed) are adjusted to ensure that a clear and complete image of the violating vehicle is captured;

[0121] The violation record method is to record the time, location, multimodal data features (such as images, sounds, sensor data), and violation type information of the violation, which can be expressed mathematically as follows:

[0122] R={t,l,F image ,F audio ,F sensor ,P violation}, where t is time, l is location, and F image is the image feature, F audio is the sound characteristic, F sensor is the sensor characteristic, P violation Is the violation type.

[0123] In this embodiment, when a violation is identified, the capture mechanism is triggered, and the capture system quickly adjusts the camera parameters (such as focal length, shutter speed, etc.) to ensure that a clear and complete image of the illegal vehicle is captured, including key information such as the front and side of the vehicle and the license plate number. Whether in a bright day or a low-light environment at night, high-quality image evidence can be obtained, providing clear and accurate visual data for subsequent traffic law enforcement. For example, when capturing a vehicle that changes lanes illegally, the process of the vehicle crossing the solid line and the license plate number can be clearly captured, making the violation clear at a glance and indisputable; at the same time, the time, place, and time of the violation are recorded. Multimodal data features (such as images, sounds, sensor data, etc.), violation types and other detailed information are uploaded to the traffic management database, and blockchain technology is used to encrypt and store the records to prevent data tampering and leakage, ensuring the security and traceability of the data; when it is necessary to inquire about the details of a violation, complete, accurate and untampered records can be retrieved from the database at any time, including multimodal information such as images and sounds at the time, which provides comprehensive and reliable data support for traffic management departments to handle accidents, punish violations and subsequent traffic flow analysis, and also facilitates car owners to verify and appeal their own violations.

[0124] The present invention also discloses a violation capture system based on image recognition, comprising:

[0125] Environmental perception module: including camera module, millimeter-wave radar module and meteorological sensor module;

[0126] The camera module is a high-resolution, high-frame-rate camera that supports low-light and wide dynamic range (HDR) imaging. The millimeter-wave radar module is a 7GHz high-frequency millimeter-wave radar, and the meteorological sensor module consists of an ultrasonic sensor, an optical raindrop sensor, and a laser scattering sensor.

[0127] Data acquisition module: obtains raw data from sensors and transmits it to the data processing center for preprocessing and feature extraction;

[0128] Data Processing Center: De-noises and enhances contrast in camera data; filters noise in millimeter-wave radar data and converts it from polar to Cartesian coordinates; normalizes meteorological data; extracts visual features (such as edges and textures) from camera data, vehicle information (such as speed, distance, and angle) from millimeter-wave radar data, and environmental features (such as light intensity and rainfall) from meteorological data; and uses a convolutional neural network (CNN) to analyze the fused data in real time and extract scene features.

[0129] Multimodal data acquisition module: This module collects multimodal data at each data collection node through cameras, microphones, millimeter-wave radars, vehicle-connected interfaces, and communication receiving devices. It then performs local processing and feature extraction on the multimodal data. It also applies homomorphic encryption and differential privacy protection to the extracted features to ensure data privacy and security. It then fuses the pre-processed multimodal data to provide a reliable foundation for subsequent violation identification and capture.

[0130] Federated learning model aggregation and fine-tuning module: The federated learning server receives encrypted feature data from each client, performs a weighted average of the local models based on the weights of the client data, and obtains a global model. The updated global model is then distributed to each client, and the client fine-tunes the global model based on the local data to adapt to the characteristics of the local traffic scenario.

[0131] Violation Recognition Module: Based on an updated federated learning model, it comprehensively analyzes multimodal data to identify vehicle violations, taking into account vehicle actions in images (such as running a red light or changing lanes across a solid line), anomalies in sound signals (such as illegal honking), driving parameter anomalies in on-board sensor data (such as speeding or sudden braking), and potential violation information in communication signals.

[0132] Capture and Recording Module: When a violation is identified, the capture mechanism is triggered, camera parameters (such as focal length and shutter speed) are adjusted to ensure a clear and complete image of the offending vehicle is captured, and the time, location, multimodal data characteristics (such as images, sounds, and sensor data), and violation type of the violation are recorded, and the information is uploaded to the database.

[0133] Database and blockchain technology support module: Use blockchain technology to encrypt and store records to prevent data tampering and leakage.

[0134] The system workflow is as follows:

[0135] The environmental perception module collects data through a variety of sensors and performs preliminary data processing and feature extraction;

[0136] The multimodal data fusion module fuses and encrypts data from different sources to ensure data security and consistency;

[0137] The deep learning analysis module uses the CNN model to perform real-time analysis on the fused data and extract scene features;

[0138] Through the federated learning model aggregation and fine-tuning module, the global model is continuously optimized to improve recognition accuracy;

[0139] The violation identification module combines multimodal data to conduct comprehensive analysis and identify violations;

[0140] When the capture and recording module detects any violations, it triggers the capture and records the relevant data;

[0141] The recorded information is encrypted and stored through database and blockchain technology to ensure the integrity and security of the data.

[0142] In summary: By combining multimodal data such as images, sounds, sensors and communication signals, the present invention enables the system to perceive and analyze traffic scenes more comprehensively, reduce the limitations of a single data source, and thus improve the accuracy of identifying violations; for example, image data can capture vehicle actions (such as running a red light, changing lanes across a solid line), sound signals can detect abnormal honking, sensor data can analyze driving parameter anomalies (such as speeding, sudden braking), and communication signals can identify potential violation information (such as inappropriate communication interactions between vehicles); the system can perceive and analyze changes in traffic scenes (such as weather, road conditions, road types) in real time etc.), and adjusts the judgment criteria and capture strategies based on scene characteristics. For example, in rainy and slippery road conditions, the system automatically adjusts the judgment criteria to improve capture accuracy. The system can make adaptive adjustments based on different traffic scenarios (such as urban roads, highways, and rural roads) and environmental conditions (such as lighting, weather, and road conditions). For example, in low-light conditions, the system automatically enhances image brightness and clarity to facilitate subsequent vehicle feature extraction and recognition. The system can make adaptive adjustments based on different traffic scenarios and environmental conditions, reducing the workload of manual intervention and system debugging, and lowering maintenance costs.

[0143] Through federated learning technology, the system can utilize data from multiple clients for joint modeling while protecting data privacy, improving the model's generalization and accuracy. The global model performs a weighted average based on the weight of each client's data to ensure the model's adaptability in different traffic scenarios. Based on the global model, the federated learning model allows clients to fine-tune it based on local data to adapt to local traffic scenarios. For example, at intersections where special traffic conditions often occur, clients can adjust the global model based on local data to more accurately identify violations at that intersection.

[0144] It uses a deep learning-based scene understanding algorithm and a convolutional neural network (CNN) for image feature extraction, which can automatically identify vehicle movements, traffic signs, road conditions and other information in traffic scenes, reducing manual intervention; when the system identifies a violation, it can automatically trigger the capture mechanism and quickly adjust the camera parameters (such as focal length, shutter speed, etc.) to ensure that clear and complete images of the violating vehicle are captured, including key information such as the front and side of the vehicle and license plate number; the system can perceive and analyze traffic scenes in real time, quickly identify violations and capture them, reducing the time for manual review and processing in traditional methods, and improving the efficiency of traffic management. The system has established a violation judgment rule library, set corresponding judgment thresholds and conditions according to different violation types, and can automatically judge violations and record relevant information, improving the system's automation level.

[0145] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for capturing violations based on image recognition, characterized in that: The following steps are involved: Environmental perception: Deploy multiple types of sensors to collect visual, vehicle speed and distance, and meteorological data; use sensor fusion methods to integrate the data to obtain environmental perception information; Scene analysis: A deep learning-based scene understanding algorithm is used to analyze perceived traffic scenes. The algorithm identifies road types, traffic signs and markings, road conditions, and dynamic changes in the surrounding environment to extract scene features. Establish a scene feature library: classify and label traffic scenes according to different scene features, and the system automatically adjusts the capture strategy; Multimodal data acquisition: Multimodal data is collected at several data collection nodes through cameras, microphones, millimeter-wave radars, vehicle networking interfaces, and communication receiving equipment. The collected data is processed and feature extracted, the extracted features are encrypted, and the processed data is fused. Server-side model aggregation: The federated learning server receives encrypted feature data from multiple clients and uses a federated learning algorithm to perform a weighted average of the local models based on the weights of the multiple client data to obtain a global model. Model distribution and fine-tuning: The updated global model is distributed to multiple clients, which then fine-tune the global model based on local data to adapt to local traffic scenarios. Violation identification: Based on the updated federated learning model, comprehensive analysis of multimodal data is performed to identify vehicle violations.

2. The method for capturing violations based on image recognition according to claim 1, characterized in that: The implementation steps of the sensor fusion method in the environmental perception are as follows: Data acquisition: Obtaining raw data from cameras, millimeter-wave radars, and weather sensors; Data preprocessing: De-noise and enhance contrast on camera data; filter noise on millimeter-wave radar data and convert it from polar coordinates to Cartesian coordinates; Normalize meteorological data; Feature extraction: extract visual features from camera data, extract vehicle information from millimeter-wave radar data; extract environmental features from meteorological data; Sensor fusion: Use Kalman filtering to fuse visual features and environmental features to obtain environmental perception information.

3. The method for capturing violations based on image recognition according to claim 1, characterized in that: The implementation steps of the scenario analysis are as follows: CNN model design: Design a convolutional neural network (CNN) model for traffic scene understanding, including multiple convolutional layers, pooling layers, and fully connected layers; Training data preparation: Collect a number of annotated traffic scene image data, including images under different weather, lighting, and road conditions; Model training: Use labeled data to train the CNN model and optimize model parameters to minimize classification error; Real-time analysis: Deploy the trained CNN model to the actual system to extract features and classify the real-time collected image data; A convolutional neural network (CNN) is used to analyze perceived traffic scenes in real time; CNN automatically extracts features from images and performs classification and recognition.

4. The method for capturing violations based on image recognition according to claim 1, characterized in that: The steps for establishing the scene feature library are as follows: Feature extraction: extracting feature vectors from the middle layers of the CNN model; Feature library construction: Classify and label traffic scenes according to different scene characteristics to build a scene feature library; use the support vector machine classification algorithm to classify scenes and determine the risk level of the current scene.

5. The method for capturing violations based on image recognition according to claim 1, characterized in that: The data preprocessing method in the multimodal data collection is as follows: Image data preprocessing: Use Gaussian filtering to remove image noise, use histogram equalization to enhance image contrast, and use perspective transformation matrix to correct image perspective distortion; Sound signal preprocessing: Use a bandpass filter to remove environmental noise interference and extract the key features of the sound signal, such as frequency and amplitude; Preprocessing of on-board sensor data and communication signal data: Perform unified unit conversion and calibration on the speed data transmitted from different vehicles, and normalize the data to a unified range.

6. The method for capturing violations based on image recognition according to claim 1, characterized in that: The federated averaging algorithm in the server-side model aggregation is: Among them, W global is the global model parameter, W i is the local model parameter of the i-th client, w i is the weight of the i-th client, N is the number of clients; The weight is determined based on data quality and data volume factors, and the mathematical model is: Among them, DataQuality i is the data quality of the i-th client.

7. The method for capturing violations based on image recognition according to claim 1, characterized in that: The steps for comprehensively analyzing multimodal data in the violation identification are as follows: The model considers vehicle motion in images, anomalies in acoustic signals, driving parameter anomalies in onboard sensor data, and potential violations in communication signals; Image data P image : Use convolutional neural networks to identify vehicle actions in images; Sound signal P audio : Apply audio processing algorithms to detect anomalies in sound signals; Vehicle sensor data P sensor :Use time series analysis method to detect driving parameter anomalies in vehicle sensor data; Communication signal P comm : Collect communication data between vehicles and between vehicles and infrastructure through communication receiving equipment to analyze potential violation information; Federated learning model: The recognition results of multimodal data are used as input and fed into the federated learning model for comprehensive analysis. The mathematical representation is as follows: P federated =Federated Learning Model(P image ,P audio ,P sensor ,P comm ), Among them, P federated It is the output of the federated learning model, indicating the identification result of the violation.

8. The method for capturing violations based on image recognition according to claim 1, characterized in that: The violation capture method also includes a judgment rule library, which sets corresponding judgment thresholds and conditions according to different violation types to determine whether there is a violation.

9. The method for capturing violations based on image recognition according to claim 1, characterized in that: The violation capture method also includes capture control and recording, which is as follows: when a violation is identified, the capture mechanism is triggered, and the capture system adjusts the camera parameters, including the front and side of the vehicle and the license plate number information, records the time, location, multimodal data characteristics, and violation type of the violation, and uploads the information to the database, and uses blockchain technology to encrypt and store the records; The camera parameters in the capture mechanism are adjusted as follows: when a violation is identified, the capture mechanism is triggered and the camera parameters are adjusted; The method of recording violations is to record the time, location, multimodal data characteristics, and violation type information of the violation.

10. A violation capture system based on image recognition, characterized in that: include: Environmental perception module: including camera module, millimeter-wave radar module and meteorological sensor module; Data acquisition module: obtains raw data from sensors and transmits it to the data processing center for preprocessing and feature extraction; Data processing center: Processes camera, millimeter-wave radar, and meteorological data; extracts visual features, vehicle information, and environmental features; uses convolutional neural networks to analyze the fused data in real time and extract scene features; Multimodal data acquisition module: collects multimodal data at each data acquisition node through cameras, microphones, millimeter-wave radars, vehicle networking interfaces, and communication receiving devices; Violation identification module: Based on the updated federated learning model, it conducts comprehensive analysis of multimodal data to identify vehicle violations.

Images