Highway License Plate Image Recognition Encryption Algorithm System Integrating Multi-Sensor Data

Abstract

This invention relates to the field of computer vision and pattern recognition technology, specifically to a highway license plate image recognition encryption algorithm system that integrates multi-sensor data. It includes a data acquisition unit, a data fusion unit, a license plate recognition unit, an encryption unit, and a cloud-based decision-making unit. The data acquisition unit acquires multi-dimensional information about vehicles, including images, speed, location, and surrounding weather data, through various sensors. The data fusion unit parallelizes the attention mechanism and feature fusion process using a GPU cluster and develops a multi-vehicle data separation algorithm based on vehicle trajectory prediction and spatial occupancy analysis. By introducing a generative adversarial network (GAN), a license plate style database is established using a large number of clear license plate images and corresponding blurred and damaged license plate images as training data. Specialized recognition sub-models are trained for different regions and types of license plates, and the appropriate sub-model is automatically selected for recognition based on the initial features of the license plate during the recognition process, reducing the false positive rate.

Description

Technical Field

[0001] This invention relates to the field of computer vision and pattern recognition technology, specifically to a highway license plate image recognition encryption algorithm system that integrates multi-sensor data. Background Technology

[0002] In highway scenarios, license plate recognition is a core module of intelligent transportation systems, widely used in electronic toll collection, traffic monitoring, and evidence collection of violations. However, traditional license plate recognition systems rely on single sensors to collect data, facing significant challenges in complex environments. For example, strong light, low light, rain, snow, and fog can cause license plate images to become blurred, reflective, or obscured, drastically reducing recognition accuracy. Multi-sensor fusion technology has gradually become a research hotspot. By integrating heterogeneous sensors such as radar, lidar, and infrared thermal imagers, multi-dimensional data can be acquired, compensating for the perception deficiencies of single sensors. In existing technologies, license plate styles vary across different regions, including color, font, and arrangement, and some license plates may have personalized decorations or modifications. This increases the difficulty of license plate recognition. Existing recognition models are insufficiently adaptable to these diverse situations, easily leading to misjudgments. Furthermore, in highway scenarios with high traffic volume and dense vehicle traffic, radar data and the visible light and infrared characteristics of different vehicles are prone to interference, making it difficult to accurately distinguish data from different vehicles.

[0003] Based on this, the present invention provides a highway license plate image recognition encryption algorithm system that integrates multi-sensor data to solve the aforementioned technical problems. Summary of the Invention

[0004] The purpose of this invention is to provide a highway license plate image recognition encryption algorithm system that integrates multi-sensor data. This invention introduces a generative adversarial network (GAN) to establish a license plate style database using a large number of clear license plate images and corresponding blurry or damaged license plate images as training data. For license plates from different regions and of different types, specialized recognition sub-models are trained. During the recognition process, the appropriate sub-model is automatically selected based on the initial features of the license plate to reduce the false positive rate. A license plate restoration generator and a discriminator are trained. The generator is responsible for restoring blurry or damaged license plate images to clear images, while the discriminator is used to determine whether the restored image is authentic. Through this adversarial training, the generator can generate high-quality restored images, providing clearer input for subsequent license plate recognition models and improving the recognition accuracy of blurry or damaged license plates.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] This invention provides a highway license plate image recognition encryption algorithm system that integrates multi-sensor data, including a data acquisition unit, a data fusion unit, a license plate recognition unit, an encryption unit, and a cloud-based decision-making unit; wherein:

[0007] The data acquisition unit is used to acquire multi-dimensional information about the vehicle, including its image, speed, location, and surrounding weather conditions, through various sensors.

[0008] The data fusion unit: parallelizes the attention mechanism and feature fusion process based on GPU clusters, and develops a multi-vehicle data separation algorithm based on vehicle trajectory prediction and spatial occupancy analysis;

[0009] The license plate recognition unit: preprocesses and repairs blurred and damaged license plate images through a generative adversarial network, while constructing a license plate style database, and automatically selects an appropriate sub-model to identify license plates of different regions and types based on preliminary license plate features;

[0010] The encryption unit encrypts the identification result using the SM4 algorithm and dynamically negotiates the key using the ECDH protocol.

[0011] The cloud-based decision-making unit is used to receive data via the DTLS protocol, and after decryption, verify the license plate and update the federated learning model.

[0012] The present invention is further configured such that: the data acquisition unit includes an image acquisition module, a speed acquisition module, a position acquisition module, and a meteorological acquisition module, wherein:

[0013] The image acquisition module is used to capture real-time image information of vehicles traveling on the highway using a high-definition camera.

[0014] The speed acquisition module uses millimeter-wave radar to detect the vehicle's speed in real time.

[0015] The location acquisition module is used to determine the specific location coordinates of a vehicle on a highway using GPS or other satellite positioning systems, enabling it to track the vehicle's driving trajectory and provide a location basis for spatial analysis of multi-vehicle data.

[0016] The meteorological acquisition module monitors the meteorological information of the vehicle's surrounding environment in real time through temperature sensors, humidity sensors, and wind speed sensors.

[0017] The present invention is further configured such that: the data fusion unit includes a spatiotemporal alignment module, a feature extraction module, a trajectory separation module, and a parallel acceleration module, wherein:

[0018] The spatiotemporal alignment module is used to achieve spatiotemporal alignment of multi-sensor data through PTP clock synchronization and Kalman filtering, and output vehicle information in a unified coordinate system.

[0019] The feature extraction module extracts vehicle and license plate features in real time based on the improved YOLOv7+CBAM network, and outputs detection boxes and feature vectors.

[0020] The trajectory separation module combines DeepSORT and LSTM to achieve multi-target tracking and trajectory prediction, and stably outputs vehicle trajectory data with IDs.

[0021] The parallel acceleration module utilizes GPU parallel computing and memory optimization technology to complete multi-source data fusion processing at high speed.

[0022] The present invention is further configured such that: the real-time extraction of vehicle and license plate features based on the improved YOLOv7+CBAM network, and the output of detection boxes and feature vectors, include the following:

[0023] A1: Based on the improved YOLOv7 network, the CBAM attention mechanism is embedded to enhance the feature response of license plate and key vehicle regions;

[0024] A2: Real-time inference at 120FPS is achieved through TensorRT acceleration, outputting vehicle detection boxes, license plate heatmaps and feature vectors, providing highly discriminative appearance and structural features for trajectory tracking;

[0025] The CBAM attention mechanism, based on channel attention weights, is calculated as follows:

[0026] M C (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))); where F∈R C×H×W The input feature map is used; the MLP is a two-layer fully connected layer with a hidden layer dimension of C / 16; σ is the Sigmoid activation function, which normalizes the weights to the [0,1] interval;

[0027] The spatial attention weights are calculated as follows:

[0028] M s (F)=σ(f 7×7 ([AvgPool(F); MaxPool(F)])); where f 7×7 The kernel is 7×7; [;] represents the channel concatenation operation;

[0029] The CIoU loss function is:

[0030] Where IoU is the intersection-over-union ratio of the predicted bounding box and the ground truth bounding box; p is the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c is the diagonal length of the minimum bounding box; and v is the aspect ratio consistency penalty term.

[0031] The TripletLoss formula is as follows:

[0032] L triplet =max(0,α+D(f)a ,f p )-D(f a ,f n )); where α is 0.3; f a The current license plate; f p For other samples with the same license plate; f n Samples for different license plates.

[0033] The present invention is further configured such that: the multi-target tracking and trajectory prediction achieved by combining DeepSORT and LSTM, and the stable output of vehicle trajectory data with IDs, includes the following:

[0034] B1: Extract vehicle appearance features based on the ReID model and predict motion state using Kalman filtering;

[0035] B2: When the probability of two vehicle trajectories intersecting exceeds 25%, spatial occupancy conflict detection is performed by fusing radar and visual confidence.

[0036] B3: Use an LSTM network to predict the trajectory for the next 3 frames, and the loss function includes a trajectory smoothness constraint term;

[0037] B4: Output a list of vehicle trajectories with unique IDs, with an ID switching rate of less than 1%.

[0038] The present invention is further configured such that: the license plate recognition unit includes an image restoration module, a style matching module, and a multi-model routing module, wherein:

[0039] The image restoration module is used to preprocess and restore blurry or damaged license plate images using a generative adversarial network.

[0040] The style matching module is used to build a database containing license plate styles of all provinces in the country, and to collect feature information on the color, font, and character arrangement of various license plates;

[0041] The multi-model routing module is used to automatically select the appropriate sub-model based on the initial features of the license plate to identify license plates from different regions and types.

[0042] The present invention is further configured such that: the construction of a database containing license plate styles of different regions and types, and the collection of feature information on the color, font, and character arrangement of various license plates, includes the following steps:

[0043] A1: Anti-interference feature extraction: Construct an anti-interference license plate feature vector using HSV main color histogram, character spacing vector and HOG contour features;

[0044] A2: Dynamic style retrieval: Uses Faiss index to quickly match the database, and prioritizes candidate styles that meet the requirements of color similarity > 85%, spacing error < 0.1 and are suitable for the modified area;

[0045] A3: Dynamic loading of sub-models: Automatically triggers dedicated models based on matching results;

[0046] A4: Conflict and Anomaly Handling: When the recognition result conflicts with the style, a secondary verification mechanism with multi-frame voting and GPS location priority is initiated.

[0047] The present invention is further configured such that: the method for automatically selecting an appropriate sub-model based on preliminary license plate features to identify license plates of different regions and types employs a three-level decision tree routing mechanism.

[0048] Primary routing: Based on the aspect ratio and main color of the license plate, it is roughly classified into blue plates, yellow plates, and new energy vehicle plates;

[0049] Secondary routing: further categorized by character spacing and font type, including military license plates and police license plates;

[0050] Three-level routing: The Triplet network is used to extract deep features for the final model selection, with a feature distance threshold θ = [0.4, 0.5].

[0051] The present invention is further configured such that: the encryption unit includes an encryption algorithm module and a key negotiation module, wherein:

[0052] The encryption algorithm module is used to encrypt the license plate recognition results using the SM4 algorithm;

[0053] The key negotiation module is used to dynamically negotiate keys using the ECDH protocol.

[0054] The present invention is further configured such that: the cloud-based decision-making unit includes a data receiving module, a license plate verification module, and a model update module, wherein:

[0055] The data receiving module is used to receive data transmitted from each unit via the DTLS protocol;

[0056] The license plate verification module is used to connect to the Ministry of Public Security's database in real time to verify the authenticity of license plates after receiving the decrypted data;

[0057] The model update module is used to update the federated learning model based on the verification results and new data.

[0058] Compared with the prior art, the beneficial effects of the present invention are:

[0059] This invention introduces a generative adversarial network (GAN) to establish a license plate style database using a large number of clear license plate images and corresponding blurry and damaged license plate images as training data. For license plates from different regions and of different types, specialized recognition sub-models are trained. During the recognition process, the appropriate sub-model is automatically selected based on the initial features of the license plate, reducing the false positive rate. A license plate restoration generator and a discriminator are trained. The generator is responsible for restoring blurry and damaged license plate images to clear images, while the discriminator is used to determine whether the restored image is authentic. Through this adversarial training, the generator can generate high-quality restored images, providing clearer input for subsequent license plate recognition models and improving the recognition accuracy of blurry and damaged license plates. Attached Figure Description

[0060] Figure 1 This is a system diagram of the highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to the present invention.

[0061] Figure 2 This is a system diagram of the data acquisition unit in the highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to the present invention.

[0062] Figure 3 This is a system diagram of the data fusion unit in the highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to the present invention.

[0063] Figure 4 This is a system diagram of the license plate recognition unit in the highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to the present invention.

[0064] Figure 5 This is a system diagram of the encryption unit in the highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to the present invention.

[0065] Figure 6 This is a system diagram of the cloud decision-making unit in the highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to the present invention.

[0066] Explanation of icon numbers:

[0067] 100. Data Acquisition Unit; 101. Image Acquisition Module; 102. Velocity Acquisition Module; 103. Location Acquisition Module; 104. Weather Acquisition Module; 200. Data Fusion Unit; 201. Spatiotemporal Alignment Module; 202. Feature Extraction Module; 203. Trajectory Separation Module; 204. Parallel Acceleration Module; 300. License Plate Recognition Unit; 301. Image Restoration Module; 302. Style Matching Module; 303. Multi-Model Routing Module; 400. Encryption Unit; 401. Encryption Algorithm Module; 402. Key Negotiation Module; 500. Cloud Decision Unit; 501. Data Receiving Module; 502. License Plate Verification Module; 503. Model Update Module. Detailed Implementation

[0068] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0069] Example:

[0070] like Figures 1-6 As shown, this embodiment provides a highway license plate image recognition encryption algorithm system that integrates multi-sensor data, including a data acquisition unit 100, a data fusion unit 200, a license plate recognition unit 300, an encryption unit 400, and a cloud decision-making unit 500; wherein: the data acquisition unit 100 is used to acquire multi-dimensional information such as vehicle images, speed, position, and surrounding environmental weather data through multiple sensors; the data fusion unit 200 performs parallel processing of the attention mechanism and feature fusion process based on a GPU cluster, and develops a multi-vehicle data separation algorithm based on vehicle trajectory prediction and spatial occupancy analysis; the license plate recognition unit 300 repairs blurred and damaged license plate images through generative adversarial networks, and simultaneously constructs a license plate style database, automatically selecting an appropriate sub-model to identify license plates of different regions and types through preliminary license plate features; the encryption unit 400 encrypts the recognition results using the SM4 algorithm and dynamically negotiates the key using the ECDH protocol; the cloud decision-making unit 500 is used to receive data through the DTLS protocol, and verify the license plate and update the federated learning model after decryption.

[0071] In this embodiment, it should be noted that the data acquisition unit 100 first simultaneously collects multi-dimensional vehicle information through a high-definition camera, millimeter-wave radar, GPS, and meteorological sensors; the data fusion unit 200 uses a GPU cluster to process data in parallel, extracts features through an improved YOLOv7+CBAM network, and combines DeepSORT and LSTM algorithms to achieve multi-vehicle trajectory prediction and spatial occupancy analysis, completing data alignment and separation; the license plate recognition unit 300 repairs damaged license plate images based on a generative adversarial network, and dynamically matches a pre-stored license plate style database through a three-level decision tree routing mechanism, calling the adaptation sub-model to complete classification and recognition; the encryption unit 400 uses the SM4 algorithm to encrypt the recognition results and dynamically negotiates keys through the ECDH protocol to ensure transmission security; finally, the cloud decision unit 500 receives encrypted data through the DTLS protocol, decrypts it to verify the authenticity of the license plate, and drives the federated learning model to iteratively update based on the verification results, forming a closed-loop intelligent recognition system with "end-edge-cloud" collaboration.

[0072] In this invention, the data acquisition unit 100 includes an image acquisition module 101, a speed acquisition module 102, a location acquisition module 103, and a weather acquisition module 104. Specifically: the image acquisition module 101 is used to capture real-time image information of vehicles traveling on the highway using a high-definition camera; the speed acquisition module 102 detects vehicle speed in real-time using millimeter-wave radar; the location acquisition module 103 uses GPS or other satellite positioning systems to determine the specific coordinates of the vehicle on the highway, enabling the tracking of the vehicle's trajectory and providing a location basis for spatial analysis of multi-vehicle data; and the weather acquisition module 104 monitors the weather information of the vehicle's surrounding environment in real-time using temperature sensors, humidity sensors, and wind speed sensors.

[0073] In this embodiment, it should be noted that the image acquisition module 101 uses a high-definition camera to capture vehicle images at a frame rate of ≥60fps, and simultaneously triggers the millimeter-wave radar of the speed acquisition module 102 to perform speed detection with an accuracy of ±0.5km / h. The position acquisition module 103 records the vehicle coordinates in real time and generates a driving trajectory through GPS or satellite dual-mode positioning with an error of <1.5m. At the same time, the environmental sensor of the weather acquisition module 104 collects temperature, humidity and wind speed data every 500ms.

[0074] In addition, it should be noted that the image information includes the vehicle's appearance and the location of the license plate, providing basic image data for subsequent license plate recognition; the meteorological information includes temperature, humidity, wind speed, and rainfall.

[0075] In this invention, the data fusion unit 200 includes a spatiotemporal alignment module 201, a feature extraction module 202, a trajectory separation module 203, and a parallel acceleration module 204. Specifically: the spatiotemporal alignment module 201 achieves spatiotemporal alignment of multi-sensor data through PTP clock synchronization and Kalman filtering, outputting vehicle information in a unified coordinate system; the feature extraction module 202 extracts vehicle and license plate features in real time based on an improved YOLOv7+CBAM network, outputting detection boxes and feature vectors; the trajectory separation module 203 combines DeepSORT and LSTM to achieve multi-target tracking and trajectory prediction, stably outputting vehicle trajectory data with IDs; and the parallel acceleration module 204... Acceleration module 204 utilizes GPU parallel computing and memory optimization techniques to perform high-speed multi-source data fusion processing; it extracts vehicle and license plate features in real time based on an improved YOLOv7+CBAM network, outputting detection boxes and feature vectors, including the following: A1: Based on the improved YOLOv7 network, a CBAM attention mechanism is embedded to enhance the feature response of key areas of license plates and vehicles; A2: Real-time inference at 120FPS is achieved through TensorRT acceleration, outputting vehicle detection boxes, license plate heatmaps, and feature vectors, providing highly discriminative appearance and structural features for trajectory tracking; the CBAM attention mechanism, based on channel attention weights, is calculated as follows:

[0076] M C (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))); where F∈R C×H×W

[0077] The input feature map is used; the MLP is a two-layer fully connected layer with a hidden layer dimension of C / 16; σ is the Sigmoid activation function, which normalizes the weights to the [0,1] interval;

[0078] Spatial attention weights are calculated as follows:

[0079] M s (F)=σ(f 7×7 ([AvgPool(F); MaxPool(F)])); where f 7×7 The kernel is 7×7; [;] represents the channel concatenation operation;

[0080] The CIoU loss function is:

[0081] Where IoU is the intersection-over-union ratio of the predicted bounding box and the ground truth bounding box; p is the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c is the diagonal length of the minimum bounding box; and v is the aspect ratio consistency penalty term.

[0082] The TripletLoss formula is:

[0083] L triplet =max(0,α+D(f) a ,f p )-D(f a ,f n )); where α is 0.3; f a The current license plate; f p For other samples with the same license plate; f n For different license plate samples, multi-target tracking and trajectory prediction are achieved by combining DeepSORT and LSTM, and stable output of vehicle trajectory data with IDs is provided, including the following: B1: Extract vehicle appearance features based on the ReID model and predict motion state through Kalman filtering; B2: When the probability of two vehicle trajectories intersecting exceeds 25%, spatial occupancy conflict detection is performed by fusing radar and visual confidence; B3: Predict the trajectory of the next 3 frames using an LSTM network, with the loss function including a trajectory smoothness constraint term; B4: Output a list of vehicle trajectories with unique IDs, with an ID switching rate of less than 1%.

[0084] In this embodiment, it should be noted that the spatiotemporal alignment module 201 first uses the PTP protocol to achieve μs-level clock synchronization, and combines Kalman filtering to compensate for spatiotemporal deviations of multiple sensors, outputting unified coordinate system data; the feature extraction module 202 extracts features in real time based on the improved YOLOv7+CBAM network, and uses TensorRT to accelerate 120FPS inference, outputting detection boxes with CIoU loss optimization; the trajectory separation module 203 fuses DeepSORT's ReID features and LSTM trajectory prediction, triggering multi-source data conflict detection when the trajectory crossover probability is >25%, ensuring that the ID switching rate is <1%; the parallel acceleration module 204 reduces the multi-source data fusion latency to less than 8ms through GPU memory optimization and CUDA kernel function parallelization. The modules form a closed loop of "spatiotemporal alignment - feature enhancement - trajectory optimization - accelerated processing", and finally output vehicle trajectory data with unique IDs, providing high-precision input for license plate recognition.

[0085] In this invention, the license plate recognition unit 300 includes an image restoration module 301, a style matching module 302, and a multi-model routing module 303. The image restoration module 301 is used to preprocess and restore blurry or damaged license plate images using a generative adversarial network. The style matching module 302 is used to construct a database containing license plate styles of different provinces across the country and to collect feature information on the color, font, and character arrangement of various license plates. The multi-model routing module 303 is used to automatically select a suitable sub-model based on the initial features of the license plate to identify license plates of different regions and types. A database containing license plate styles from different regions and types is constructed, collecting feature information on the color, font, and character arrangement of various license plates. This includes the following steps: A1: Anti-interference feature extraction: Constructing anti-interference license plate feature vectors using HSV primary color histograms, character spacing vectors, and HOG contour features; A2: Dynamic style retrieval: Using Faiss indexing for fast database matching, prioritizing candidate styles that meet the requirements of color similarity > 85%, spacing error < 0.1, and suitability for modified areas; A3: Dynamic loading of sub-models: Automatically triggering dedicated models based on matching results; A4: Conflict and anomaly handling: When the recognition result conflicts with the style, a secondary verification mechanism based on multi-frame voting and GPS location priority is initiated. To automatically select a suitable sub-model based on the initial features of the license plate to identify license plates from different regions and types, a three-level decision tree routing mechanism is adopted: Level 1 routing: coarse classification based on the aspect ratio and main color of the license plate, including blue plates, yellow plates, and new energy plates; Level 2 routing: fine classification based on character spacing and font type, including military plates and police plates; Level 3 routing: using a Triplet network to extract deep features for final model selection, with a feature distance threshold θ = [0.4, 0.5].

[0086] In this embodiment, it should be noted that the image restoration module 301 first uses a generative adversarial network generator with 8 residual blocks, and the discriminator uses a PatchGAN structure to restore blurred and damaged license plates. Through joint optimization of perceptual loss and adversarial loss, the PSNR index is improved by more than 12dB. The style matching module 302 constructs a license plate feature database covering all provinces in China. It generates anti-interference feature vectors by 16-bin quantization of HSV primary color histogram, character spacing vector normalization error <0.1, and HOG feature 9-directional gradient, and based on Fais The s-index enables millisecond-level style retrieval, filtering candidate styles that meet the criteria of color similarity > 85% and are suitable for the modified area. The multi-model routing module 303 uses a three-level decision tree mechanism to dynamically select sub-models: the first-level routing uses a coarse classification based on the aspect ratio of blue license plates (440×140mm ± 5%) and the RGB value threshold for the main color; the second-level routing uses character spacing (e.g., the spacing between the second and third characters of a military license plate is 34mm) and the font feature of stroke width ratio for fine classification; the third-level routing uses a Triplet network to calculate the depth feature distance, with a threshold θ = [0.4, 0.5] to complete the final matching. When a conflict is identified, multi-frame voting is triggered, including ≥3 consistent frames out of 5 consecutive frames and GPS location priority verification based on administrative division codes for secondary verification, achieving a recognition accuracy of ≥ 98.5% in complex scenarios, tested at a vehicle speed of 100km / h.

[0087] Furthermore, it should be noted that the preprocessing and restoration of blurred and damaged license plate images using generative adversarial networks includes the following: a generator with an encoder-decoder structure is used, embedding a ConvLSTM module to model the temporal features of the license plate, and combining a spatial attention mechanism to enhance the reconstruction capability of locally damaged areas; pixel-level L1 loss, structural similarity SSIM loss, and adversarial loss are fused, with the formula: L total =λ1·L L1 +λ2·L SSIM +λ3·L GAN Where λ1=0.8, λ2=0.6, λ3=1.0, the weight allocation was optimized through experiments.

[0088] In this invention, the encryption unit 400 includes an encryption algorithm module 401 and a key negotiation module 402, wherein: the encryption algorithm module 401 is used to encrypt the license plate recognition result using the SM4 algorithm; and the key negotiation module 402 is used to dynamically negotiate the key using the ECDH protocol.

[0089] In this embodiment, it should be noted that the encryption algorithm module 401 uses the national cryptographic SM4 algorithm to encrypt the license plate recognition results. Data confidentiality is ensured through 32 rounds of round function iteration, with a measured encryption throughput of 1.2Gbps. The key negotiation module 402 dynamically generates session keys based on the ECDH protocol. During the negotiation process, temporary keys and two-way authentication are used to prevent man-in-the-middle attacks. The key update cycle is set to 24 hours or automatically triggers renegotiation when the data volume of a single session exceeds 1GB. The two modules transmit key materials through a secure memory area. The encrypted data packets are appended with timestamps and sequence numbers to form TLV format messages, which are finally transmitted to the cloud via the DTLS 1.3 protocol, achieving an end-to-end encryption latency of less than 15ms and a security system resistant to quantum computing attacks.

[0090] In addition, it should be noted that the national standard SM4 algorithm is a 128-bit block cipher in CBC mode with PKCS#7 padding; the ECDH protocol key length is 256 bits.

[0091] In this invention, the cloud-based decision-making unit 500 includes a data receiving module 501, a license plate verification module 502, and a model update module 503. Specifically: the data receiving module 501 is used to receive data transmitted from various units via the DTLS protocol; the license plate verification module 502 is used to verify the authenticity of license plates in real time by connecting to the Ministry of Public Security's database after receiving the decrypted data; and the model update module 503 is used to update the federated learning model based on the verification results and new data.

[0092] In this embodiment, it should be noted that the data receiving module 501 receives encrypted data streams in real time via the DTLS 1.3 protocol and uses a double-buffered queue mechanism to handle network jitter, ensuring that 99.9% of data packets are received completely under 100Mbps bandwidth. The license plate verification module 502 connects to the Ministry of Public Security's vehicle database and improves the efficiency of high-frequency queries by 3 times through multi-level caching, including a Redis cluster and a local LRU cache. It also introduces blockchain notarization technology to record the verification results in an immutable manner. The model update module 503 adopts a federated learning framework, specifically the FATE framework. It automatically labels the data confidence level based on the verification results. When the license plate misidentification rate within a 50-meter radius centered on the GPS coordinates exceeds the threshold (e.g., 5%) three times consecutively, an incremental update mechanism is triggered. The model version is then pushed to the edge nodes in a grayscale manner after digital signature verification.

[0093] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0094] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A highway license plate image recognition encryption algorithm system integrating multi-sensor data, characterized in that, It includes a data acquisition unit (100), a data fusion unit (200), a license plate recognition unit (300), an encryption unit (400), and a cloud-based decision-making unit (500); among which: The data acquisition unit (100) is used to acquire multi-dimensional information about the vehicle's image, speed, position, and surrounding weather through multiple sensors. The data fusion unit (200) parallelizes the attention mechanism and feature fusion process based on the GPU cluster, and develops a multi-vehicle data separation algorithm based on vehicle trajectory prediction and spatial occupancy analysis. The license plate recognition unit (300) uses a generative adversarial network to preprocess and repair blurry or damaged license plate images, while simultaneously constructing a license plate style database. It also automatically selects an appropriate sub-model based on the initial features of the license plate to identify license plates from different regions and of different types. The encryption unit (400) encrypts the identification result using the SM4 algorithm and dynamically negotiates the key using the ECDH protocol. The cloud-based decision unit (500) is used to receive data via the DTLS protocol, and after decryption, verify the license plate and update the federated learning model. The data fusion unit (200) includes a spatiotemporal alignment module (201), a feature extraction module (202), a trajectory separation module (203), and a parallel acceleration module (204), wherein: The spatiotemporal alignment module (201) is used to achieve spatiotemporal alignment of multi-sensor data through PTP clock synchronization and Kalman filtering, and output vehicle information in a unified coordinate system. The feature extraction module (202) extracts vehicle and license plate features in real time based on the improved YOLOv7+CBAM network and outputs detection boxes and feature vectors. The trajectory separation module (203) combines DeepSORT and LSTM to achieve multi-target tracking and trajectory prediction, and stably outputs vehicle trajectory data with ID; The parallel acceleration module (204) utilizes GPU parallel computing and memory optimization technology to complete multi-source data fusion processing at high speed; The license plate recognition unit (300) includes an image restoration module (301), a style matching module (302), and a multi-model routing module (303), wherein: The image restoration module (301) is used to preprocess and restore blurry or damaged license plate images using a generative adversarial network; The style matching module (302) is used to build a database containing license plate styles of all provinces in the country and to collect feature information of the color, font and character arrangement of various license plates; The multi-model routing module (303) is used to automatically select the appropriate sub-model based on the initial features of the license plate to identify license plates from different regions and types.

2. The highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to claim 1, characterized in that, The data acquisition unit (100) includes an image acquisition module (101), a speed acquisition module (102), a position acquisition module (103), and a weather acquisition module (104), wherein: The image acquisition module (101) is used to capture image information of vehicles traveling on the highway in real time using a high-definition camera; The speed acquisition module (102) detects the vehicle speed in real time using millimeter-wave radar; The location acquisition module (103) is used to determine the specific location coordinates of the vehicle on the highway using GPS or other satellite positioning systems, and can track the vehicle's driving trajectory. It also provides a location basis for the spatial analysis of multi-vehicle data. The meteorological acquisition module (104) monitors the meteorological information of the vehicle’s surrounding environment in real time through temperature sensors, humidity sensors, and wind speed sensors.

3. The highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to claim 1, characterized in that, The improved YOLOv7+CBAM network is used to extract vehicle and license plate features in real time, and output detection boxes and feature vectors. Includes the following: A1: Based on the improved YOLOv7 network, the CBAM attention mechanism is embedded to enhance the feature response of license plate and key vehicle regions; A2: Real-time inference at 120FPS is achieved through TensorRT acceleration, outputting vehicle detection boxes, license plate heatmaps and feature vectors, providing highly discriminative appearance and structural features for trajectory tracking; The CBAM attention mechanism, based on channel attention weights, is calculated as follows: ;in, The input feature map is used; the MLP consists of two fully connected layers with a hidden layer dimension of C / 16. The Sigmoid activation function normalizes the weights to the [0,1] interval; Spatial attention weights are calculated as follows: ;in, for Convolution kernel; [;] represents channel concatenation operation; The CIoU loss function is: ;in, is the intersection-over-union ratio (IoU) of the predicted bounding box and the ground truth bounding box; P is the Euclidean distance between the center points of the predicted and ground truth bounding boxes; C is the diagonal length of the minimum bounding box; v is the aspect ratio consistency penalty term. ; The formula for Triplet Loss is: ;in, It is 0.3; The current license plate; Other samples with the same license plate; Samples for different license plates.

4. The highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to claim 1, characterized in that, The method combines DeepSORT and LSTM to achieve multi-target tracking and trajectory prediction, and stably outputs vehicle trajectory data with IDs, including the following: B1: Extract vehicle appearance features based on the ReID model and predict motion state using Kalman filtering; B2: When the probability of two vehicle trajectories intersecting exceeds 25%, spatial occupancy conflict detection is performed by fusing radar and visual confidence. B3: Use an LSTM network to predict the trajectory for the next 3 frames, and the loss function includes a trajectory smoothness constraint term; B4: Output a list of vehicle trajectories with unique IDs, with an ID switching rate of less than 1%.

5. The highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to claim 1, characterized in that, The project involves constructing a database containing license plate styles from all provinces across the country, collecting feature information on the color, font, and character arrangement of various license plates. Includes the following steps: A1: Anti-interference feature extraction: Construct an anti-interference license plate feature vector using HSV main color histogram, character spacing vector and HOG contour features; A2: Dynamic style retrieval: Uses Faiss index to quickly match the database and filter candidate styles that meet the requirements of color similarity > 85%, spacing error < 0.1, and are suitable for the modified area; A3: Dynamic loading of sub-models: Automatically triggers dedicated models based on matching results; A4: Conflict and Anomaly Handling: When the recognition result conflicts with the style, a secondary verification mechanism with multi-frame voting and GPS location priority is initiated.

6. The highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to claim 1, characterized in that, The method for automatically selecting a suitable sub-model based on preliminary license plate features to identify license plates from different regions and types employs a three-level decision tree routing mechanism. Primary routing: Based on the aspect ratio and main color of the license plate, it is roughly classified into blue plates, yellow plates, and new energy vehicle plates; Secondary routing: further categorized by character spacing and font type, including military license plates and police license plates; Three-level routing: Using a Triplet network to extract deep features for final model selection, feature distance threshold. .

7. The highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to claim 1, characterized in that, The encryption unit (400) includes an encryption algorithm module (401) and a key negotiation module (402), wherein: The encryption algorithm module (401) is used to encrypt the license plate recognition result using the SM4 algorithm; The key negotiation module (402) is used to dynamically negotiate keys using the ECDH protocol.

8. The highway license plate image recognition encryption algorithm system that integrates multi-sensor data according to claim 1, characterized in that, The cloud-based decision-making unit (500) includes a data receiving module (501), a license plate verification module (502), and a model update module (503), wherein: The data receiving module (501) is used to receive data transmitted from each unit via the DTLS protocol; The license plate verification module (502) is used to connect to the Ministry of Public Security's database in real time to verify the authenticity of the license plate after receiving the decrypted data; The model update module (503) is used to update the federated learning model based on the verification results and new data.

Images