System and method for identifying and detecting dynamic and static spilled objects on expressway

Abstract

The invention discloses a system and a method for identifying and detecting dynamic and static spilled objects on an expressway, and belongs to the technical field of expressway detection. In order to more accurately identify dynamic and static thrown objects on an expressway, a thunder-vision all-in-one machine is respectively connected with a video identification module and a radar identification module, the video identification module and the radar identification module are respectively connected with a thrown object identification module, and the thrown object identification module is connected with a thrown object vehicle information identification module. The thrown object vehicle information identification module is connected with the upper computer; the video recognition module is used for detecting an abnormal object image on the expressway and an abnormal vehicle lane changing behavior image on the expressway; the radar identification module is used for detecting point cloud data of an abnormal object on a highway; the throwing object identification module is used for integrating the output of the video identification module and the radar identification module and judging and identifying suspected throwing objects; the thrown object vehicle information identification module is used for identifying video clips, license plate information or RFID information of thrown object vehicles.

Description

Technical Field

[0001] The invention belongs to the technical field of highway detection, and in particular relates to a system and method for identifying and detecting dynamic and static scattered objects on a highway. Background Art

[0002] Highway spilled materials refer to objects that are scattered or dropped by the vehicle in front or fly from the opposite lane while vehicles are driving on the highway, such as stones, fallen goods, beverage bottles, waste cardboard boxes, waste tires, and even tarps, which are often scattered or discarded on the highway road surface.

[0003] Therefore, timely and accurate detection of road spills is an important requirement for video traffic monitoring equipment, which plays an important role in the prevention, investigation and treatment of traffic hazards, the maintenance of the traffic environment, and the recording and alarm of traffic incidents.

[0004] Therefore, it is crucial to quickly identify and analyze the spilled objects on the highway. Common highway spilled objects include semi-trailer tarpaulins, express boxes, paper, tires, plastic products, etc. Due to the differences in the possible characteristics of different types of spilled objects, the accuracy and timeliness of the identification will be affected. At the same time, the different trajectories of the objects may have different degrees of impact on the vehicles behind. In order to ensure the safety and supervision of the highway, it is necessary to propose an identification rule and detection method to provide a more accurate reference for the relevant departments.

[0005] At present, the identification of spilled objects on highways is mainly based on video recognition technology. However, when detecting spilled objects based on video images, it is inevitably affected by the following factors: lighting, shadows, occlusions, camera shake, low contrast, similarity between foreground and background, dynamic background changes and image blur. Static foreground object detection algorithms, most of these algorithms need to be based on a reliable and real background. At the same time, since the common on-site supervision methods of traffic police cannot be applied to highways in a large area, and there are few electric police with vehicle position determination and monitoring functions, or they must be equipped with field speed measurement equipment such as radar, the existing radar-visual integrated monitoring equipment can rely on the roadside mounted camera and driving route detection algorithm, and adopt a recognition algorithm combining vision and radar. When the vehicle that spills objects changes lanes, it retains the illegal video evidence for capture and pushes it to the section company and road administration law enforcement department as law enforcement record, which can effectively punish the illegal spilling vehicles. Summary of the invention

[0006] The problem to be solved by the present invention is to more accurately identify dynamic and static scattered objects on a highway, and a dynamic and static scattered object identification and detection system and detection method on a highway are proposed.

[0007] To achieve the above object, the present invention is implemented through the following technical solutions:

[0008] A system for identifying and detecting dynamic and static spilled objects on a highway, comprising a radar-visual integrated machine, a video identification module, a radar identification module, a spilled object identification module, a spilled object vehicle information identification module, and a host computer;

[0009] The integrated radar and vision machine is respectively connected to a video recognition module and a radar recognition module, the video recognition module and the radar recognition module are respectively connected to a spilled object recognition module, the spilled object recognition module is connected to a spilled object vehicle information recognition module, and the spilled object vehicle information recognition module is connected to a host computer;

[0010] The camera and radar of the integrated radar and vision machine respectively collect real-time images on the highway, and the network interfaces of the camera, radar and edge computing gateway are connected via Ethernet cables;

[0011] The video recognition module is used to detect images of abnormal objects on the highway and images of abnormal vehicle lane change behavior on the highway;

[0012] The radar recognition module is used to detect point cloud data of abnormal objects on the highway;

[0013] The spilled object recognition module is used to integrate the outputs of the video recognition module and the radar recognition module to identify suspected spilled objects;

[0014] The object-scattering vehicle information identification module is used to identify the video clip, license plate information or RFID information of the object-scattering vehicle;

[0015] The host computer receives the video clip, license plate information or RFID information of the object-scattering vehicle obtained by the object-scattering vehicle information identification module.

[0016] Furthermore, the edge computing gateway accesses the real-time video stream information collected by the camera of the RayVision integrated machine through the RTSP video stream address.

[0017] A method for identifying and detecting dynamic and static scattered objects on a highway is implemented according to a system for identifying and detecting dynamic and static scattered objects on a highway, and comprises the following steps:

[0018] S1. The camera and radar of the integrated machine collect real-time images on the highway and transmit them to the video recognition module and radar recognition module respectively;

[0019] S2. The video recognition module determines abnormal vehicle lane change behavior on the video image obtained in step S1, then preprocesses the video image obtained in step S1, then performs image background modeling to obtain a road surface image, and then uses the trained AI target detection model to detect the road surface image and identify the output image of the video recognition module;

[0020] S3. The radar recognition module preprocesses the radar image obtained in step S1, then performs image background modeling to obtain a processed radar image, and then uses the trained 3D target detection model to perform target detection and recognition on the processed radar data to obtain an output image of the radar recognition module;

[0021] S4. The video recognition module output image obtained in step S2 and the radar recognition module output image obtained in step S3 are input into the spilled object recognition module for integration, and the suspected spilled objects are judged, identified and classified to obtain an output image for identifying the spilled objects;

[0022] S5. The output image obtained in step S4 for identifying the scattered objects is input into the scattered object vehicle information recognition module to extract the video clip, license plate information or RFID information of the scattered object vehicle;

[0023] S6. The video clip, license plate information or RFID information of the vehicle that spilled the objects extracted in step S5 is transmitted to the road section company and the road administration law enforcement department, prompting them to handle the dynamic and static spilled objects on the highway.

[0024] Furthermore, the specific implementation method of step S2 includes the following steps:

[0025] S2.1. The video recognition module collects the vehicle trajectory data T from the video image obtained in step S1, and the expression is:

[0026]

[0027] Where N is the number of data points, (x i ,y i ) is the trajectory coordinate of the i-th data point;

[0028] The trajectory is smoothed by a Gaussian filter to obtain the smoothed trajectory T′, which is expressed as:

[0029] T′=GaussianFilter(T,σ)

[0030] Where σ is the standard deviation of the Gaussian filter, and then the trajectory is divided into k segments of fixed length L, and the expression is:

[0031]

[0032] Then calculate the velocity v on the segmented trajectory i , acceleration a i and the rate of change of the angular direction Δθ i , the expression is:

[0033]

[0034] Δθ i =θ i+1 -θ i

[0035] in, is the direction angle. When one or more of the speed, acceleration and direction angle change rate is greater than a threshold, it is considered that an abnormal lane change behavior exists, and a video image of the abnormal lane change behavior is obtained;

[0036] S2.2. The video recognition module preprocesses the video image obtained in step S1, including image distortion correction, adaptive illumination enhancement, and expansion of the training sample set, to obtain a preprocessed video image;

[0037] S2.3. Perform image background modeling on the preprocessed video image obtained in step S2.2, and use the Gaussian mixture model GMM model suitable for dynamic backgrounds to remove moving targets including vehicles and pedestrians, retain the road surface area as the area to be detected, and obtain a road surface image;

[0038] S2.4. Train the YOLOv5 model using an image dataset containing various road spills, obtain the trained YOLOv5 model as an AI target detection model, and use the trained AI target detection model to detect the road image obtained in step S2.3 to obtain the output image detected by the video recognition module;

[0039] S2.5. The video image of abnormal lane change behavior obtained in step S2.1 is used to generate a trajectory distribution map based on the vehicle trajectory data of the past period of time. The value of each pixel in the trajectory distribution map represents the total duration of the trajectory at that point in unit time. The distribution map is input into the feature extraction network of the multi-layer convolution layer to obtain the vehicle trajectory embedding vector, and then the vehicle trajectory embedding vector is spliced ​​with the video recognition module detection output image obtained in step S2.3 to obtain the video recognition module output image, which is expressed as:

[0040] Input=Concat(Conv(TrajectoryMap), VideoDetector(Image))

[0041] Among them, TrajectoryMap is the trajectory distribution map, Conv is the multi-layer convolution layer, Image is the original image data, VideoDetector is the detection model in the video recognition module, and Input is the output image of the video recognition module.

[0042] Furthermore, the method for constructing the Gaussian mixture model GMM model suitable for dynamic background in step S2.3 includes the following steps:

[0043] S2.3.1. Set the color value of each pixel to be described by multiple Gaussian distributions. Assuming that each pixel value is X, the expression of the Gaussian mixture model P(X) is:

[0044]

[0045] Where K is the number of Gaussian distributions, w i is the weight of the ith Gaussian distribution, μ i is the mean vector of the i-th Gaussian distribution, ∑ i is the covariance matrix of the ith Gaussian distribution, The mean is μ i The covariance matrix is ​​∑ i Gaussian distribution of

[0046] S2.3.2. For each new pixel value X t , the Gaussian mixture model performs the following update steps:

[0047] Calculate pixel value X t The distance d from each Gaussian distribution i , the expression is:

[0048] d i =||X t -μ i ||

[0049] Then determine the current pixel value X t Whether it matches a Gaussian distribution, if d i If it is less than a certain threshold, usually 2.5σ, then X is considered t Belongs to the i-th Gaussian distribution;

[0050] For the matching Gaussian distribution Update the parameters and the expression for updating the mean vector is:

[0051] μ i =(1-α)μ i +αX t

[0052] Update the covariance matrix∑ i The expression is:

[0053] ∑ i =(1-α)∑ i +α(X t -μ i )(X t -μ i ) T

[0054] Update weight w i The expression is:

[0055] w i =(1-α)w i +α

[0056] For mismatched Gaussian distribution, the expression for weight decay is:

[0057] w i =(1-α)w i

[0058] If no Gaussian distribution matches the current pixel value X t , then use X t Create a new Gaussian distribution and delete the Gaussian distribution with the smallest weight;

[0059] S2.3.3. In order to ensure that the sum of all weights is 1, all weights are normalized, and the expression is:

[0060]

[0061] For each pixel, if the current pixel value X t cannot be described by any Gaussian distribution, that is, d i >2.5σ, the pixel is considered to belong to the foreground image.

[0062] Furthermore, the specific implementation method of step S3 includes the following steps:

[0063] S3.1. The radar recognition module preprocesses the radar image obtained in step S1 by performing denoising processing using the surface outlier removal SOR method and median filtering to obtain a preprocessed radar image;

[0064] S3.2. Perform background modeling on the preprocessed radar image obtained in step S3.1, first voxelize the radar point cloud data, select a grid size, distribute the points to prime voxels, and then perform historical statistical modeling on the spatial voxels by using a Gaussian mixture model (GMM) model suitable for dynamic backgrounds, adjust the parameters of the GMM model, including the number of Gaussian distributions and the covariance matrix, remove moving targets including vehicles and pedestrians, retain static background information data, and obtain a processed radar image;

[0065] S3.3. Use the trained 3D target detection model to perform target detection and recognition on the processed radar data obtained in step S3.2. Use the PointPillars network to adjust the learning rate, batch size, and number of training rounds to obtain a model with good detection capabilities to detect 3D targets on the ground and obtain the output image of the radar recognition module.

[0066] Furthermore, the specific implementation method of step S4 includes the following steps:

[0067] S4.1. Input the output image of the video recognition module obtained in step S2 into the AI ​​classification model in the spilled object recognition module, and use the modified ResNet-50 model as the AI ​​classification model. The ResNet-50 model structure is as follows:

[0068] ResNet-50=Dense(AvgPooling(Conv5_x(Conv4_x(Conv3_x(Conv2_x(Conv1(Input)))))))

[0069] Among them, AvgPooling is the average pooling layer, Conv1, Conv2_x, Conv3_x, Conv4_x, Conv5_x are the convolution modules of the first layer depth, the convolution modules of the second layer depth, the convolution modules of the third layer depth, the convolution modules of the fourth layer depth, and the convolution modules of the fifth layer depth respectively, and Dense is the classification result output by the fully connected layer after the SoftMax activation function;

[0070] S4.2. Input the radar recognition module output image obtained in step S3 into the deep learning-based point cloud classification model PointNet in the spilled object recognition module, and perform secondary classification on the target suspected to be a spilled object to determine whether the target is a spilled object. The expression of PointNet is:

[0071] PointNet=SoftMax(MaxPooling(SoftMax(T-Nett(MLP(T-Net(PointCloud))))))

[0072] Among them, T-Net is the input transformation network used to learn the transformation matrix of the input point cloud, MLP is a multi-layer shared multi-layer perceptron, PointCloud is the point cloud data, and MaxPooling is the maximum pooling layer; finally, SoftMax is used to predict the final classification score, and the expression is:

[0073] T-Net(Input)=MLP k×n (Input)×MLP n×n (MLP k×n (Input));

[0074] S4.3. Use the multi-frame image judgment method to make a comprehensive judgment on the video detection results obtained in step S4.1 and the radar classification results obtained in step S4.2. If the video classification results at multiple moments are consistent with the radar detection classification results, the spilled object detection results are considered credible. If the image classification results at multiple moments are inconsistent with the classification results, the detection results are considered unreliable and will be eliminated. If it is determined to be spilled objects, a rectangular frame will be used to mark the spilled objects on the road surface, and an abnormal road event alarm message will be prompted in the system. The information will be stored in the database, and the vehicle data for a period of time before and after the spilled objects will be retrieved and input into the spilled object vehicle information recognition module.

[0075] Furthermore, the specific implementation method of step S5 includes the following steps:

[0076] S5.1. The output image of the scattered object obtained in step S4 is input into the scattered object vehicle information recognition module. When the scattered object recognition module recognizes the scattered object information, firstly, according to the recognition timestamp, the scattered vehicle ID is traced from the historical vehicle identification information according to the location, trajectory and other information, and the video recording is started to intercept the video clips before and after the scattering behavior for a certain time;

[0077] S5.2. Detect and crop the scattered vehicle image from the captured video image using the deep learning-based vehicle detection model Yolov5;

[0078] The backbone network of YOLOv5 adopts a lightweight model architecture. CSPDarknet53 and EfficientNet are selected as the basic networks, which are responsible for extracting features from the input image. The feature vectors are then used for further processing and fusion of feature maps through the Neck structure. Finally, the detection head is responsible for generating prediction boxes at different scales and predicting the category probability and bounding box position of each box.

[0079] The scattered vehicle images are detected using the EfficientDet-based license plate detection model. If the license plate can be detected, the license plate image is cropped out, the perspective transformation matrix of the license plate is calculated based on the detected license plate corner point data, and the license plate image is projected into a front view. The license plate information is extracted using the license plate recognition PaddleOCR model; if the license plate cannot be detected, the RFID information of the scattered vehicle is recorded.

[0080] Furthermore, the step S5.2 of projecting the license plate is as follows: first, obtaining the corner points of the license plate P = {(x1, y1), (x2, y2), (x3, y3), (x4, y4)};

[0081] Then calculate the perspective transformation matrix M, the expression is:

[0082] M=getPerspectiveTransform(P,P′)

[0083] Among them, getPerspectiveTransform is used to calculate the perspective transformation matrix, P is the corner point of the license plate in the image, and P′ is the corner point of the license plate in the plane rectangular coordinate system established according to the actual size;

[0084] The license plate image is transformed into a perspective view to obtain the front view I′, which is expressed as:

[0085] I′=warpPerspective(I,M)

[0086] Among them, I is the actual image of the license plate, and I′ is the front view of the license plate.

[0087] Beneficial effects of the present invention:

[0088] The method for identifying and detecting dynamic and static scattered objects on a highway described in the present invention is combined with the edge side of a radar camera integrated machine deployed on the roadside, and uses edge computing, deep learning, and machine vision technology to perform image recognition and analysis on the real-time video stream of a high-definition camera, and uses a recognition algorithm of a multimodal pre-trained large model to quickly report the detection results to the road section company and the road administration law enforcement department platform. It can analyze and classify the scattered objects, and has the characteristics of high economy, rapid detection, and timely uploading of illegal vehicle information, thereby ensuring the safe operation of the highway. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 It is a structural schematic diagram of a system for identifying and detecting dynamic and static spilled objects on a highway according to the present invention;

[0090] Figure 2 It is a flow chart of a method for identifying and detecting dynamic and static scattered objects on a highway according to the present invention;

[0091] Figure 3 It is a schematic diagram of the framework structure of a method for identifying and detecting dynamic and static scattered objects on a highway according to the present invention;

[0092] Figure 4 This is a schematic diagram of image background modeling of the present invention;

[0093] Figure 5 This is a schematic diagram of an output image for identifying spilled objects according to the present invention. DETAILED DESCRIPTION

[0094] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0095] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0096] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Figure 5 The detailed instructions are as follows:

[0097] Embodiment 1:

[0098] A system for identifying and detecting dynamic and static spilled objects on a highway, comprising a radar-visual integrated machine, a video identification module, a radar identification module, a spilled object identification module, a spilled object vehicle information identification module, and a host computer;

[0099] The integrated radar and vision machine is respectively connected to a video recognition module and a radar recognition module, the video recognition module and the radar recognition module are respectively connected to a spilled object recognition module, the spilled object recognition module is connected to a spilled object vehicle information recognition module, and the spilled object vehicle information recognition module is connected to a host computer;

[0100] The camera and radar of the integrated radar and vision machine respectively collect real-time images on the highway, and the network interfaces of the camera, radar and edge computing gateway are connected via Ethernet cables;

[0101] The video recognition module is used to detect images of abnormal objects on the highway and images of abnormal vehicle lane change behavior on the highway;

[0102] The radar recognition module is used to detect point cloud data of abnormal objects on the highway;

[0103] The spilled object recognition module is used to integrate the outputs of the video recognition module and the radar recognition module to identify suspected spilled objects;

[0104] The object-scattering vehicle information identification module is used to identify the video clip, license plate information or RFID information of the object-scattering vehicle;

[0105] The host computer receives the video clip, license plate information or RFID information of the object-scattering vehicle obtained by the object-scattering vehicle information identification module.

[0106] Furthermore, the edge computing gateway accesses the real-time video stream information collected by the camera of the RayVision integrated machine through the RTSP video stream address.

[0107] Embodiment 2:

[0108] A method for identifying and detecting dynamic and static scattered objects on a highway is implemented according to a system for identifying and detecting dynamic and static scattered objects on a highway described in Example 1, and is characterized by comprising the following steps:

[0109] S1. The camera and radar of the integrated machine collect real-time images on the highway and transmit them to the video recognition module and radar recognition module respectively;

[0110] S2. The video recognition module determines abnormal vehicle lane change behavior on the video image obtained in step S1, then preprocesses the video image obtained in step S1, then performs image background modeling to obtain a road surface image, and then uses the trained AI target detection model to detect the road surface image and identify the output image of the video recognition module;

[0111] Furthermore, the specific implementation method of step S2 includes the following steps:

[0112] S2.1. The video recognition module collects the vehicle trajectory data T from the video image obtained in step S1, and the expression is:

[0113]

[0114] Where NJ is the number of data points, (x i ,y i ) is the trajectory coordinate of the i-th data point;

[0115] The trajectory is smoothed by a Gaussian filter to obtain the smoothed trajectory T′, which is expressed as:

[0116] T′=GaussianFilter(T,σ)

[0117] Where σ is the standard deviation of the Gaussian filter, and then the trajectory is divided into k segments of fixed length L, and the expression is:

[0118]

[0119] Then calculate the velocity v on the segmented trajectory i , acceleration a i and the rate of change of the angular direction Δθ i , the expression is:

[0120]

[0121] Δθ i =θ i+1 -θ i

[0122] in, is the direction angle. When one or more of the speed, acceleration and direction angle change rate is greater than a threshold, it is considered that an abnormal lane change behavior exists, and a video image of the abnormal lane change behavior is obtained;

[0123] S2.2. The video recognition module preprocesses the video image obtained in step S1, including image distortion correction, adaptive illumination enhancement, and expansion of the training sample set, to obtain a preprocessed video image;

[0124] Image distortion correction is to eliminate camera image distortion, making the road surface area in the image clearer and more regular, which is convenient for subsequent background modeling and target detection. Adaptive illumination enhancement is to improve the contrast and brightness of the image, making the objects in the image more prominent and reducing the impact of illumination changes on image quality.

[0125] S2.3. Perform image background modeling on the preprocessed video image obtained in step S2.2, and use the Gaussian mixture model GMM model suitable for dynamic backgrounds to remove moving targets including vehicles and pedestrians, retain the road surface area as the area to be detected, and obtain a road surface image;

[0126] The GMM model is a probability-based background modeling method. It assumes that the grayscale value of each pixel follows a mixed Gaussian distribution. By learning the historical grayscale value of each pixel, the parameters of the Gaussian component of each pixel are obtained. Then, according to the current grayscale value of each pixel and the parameters of the Gaussian component, it is determined whether the pixel belongs to the background. The GMM model can adapt to the lighting changes, shadows, camera shaking and other conditions under the dynamic background, and effectively separate the static background and dynamic foreground. In this method, we regard the static background as the road surface area and the dynamic foreground as moving targets such as vehicles and pedestrians, remove them from the video image, and only retain the road surface area as the area to be detected.

[0127] Furthermore, the method for constructing the Gaussian mixture model GMM model suitable for dynamic background in step S2.3 includes the following steps:

[0128] S2.3.1. Set the color value of each pixel to be described by multiple Gaussian distributions. Assuming that each pixel value is X, the expression of the Gaussian mixture model P(X) is:

[0129]

[0130] Where K is the number of Gaussian distributions, w i is the weight of the ith Gaussian distribution, μ i is the mean vector of the i-th Gaussian distribution, ∑ i is the covariance matrix of the ith Gaussian distribution, The mean is μ i The covariance matrix is ​​∑ i Gaussian distribution of

[0131] S2.3.2. For each new pixel value X t , the Gaussian mixture model performs the following update steps:

[0132] Calculate pixel value X t The distance d from each Gaussian distribution i , the expression is:

[0133] d i =||X t -μ i ||

[0134] Then determine the current pixel value X t Whether it matches a Gaussian distribution, if d i If it is less than a certain threshold, usually 2.5σ, then X is considered t Belongs to the i-th Gaussian distribution;

[0135] For the matching Gaussian distribution Update the parameters and the expression for updating the mean vector is:

[0136] μ i =(1-α)μ i +αX t

[0137] Update the covariance matrix∑ i The expression is:

[0138] ∑ i =(1-α)∑ i +α(X t -μ i )(X t -μ i ) T

[0139] Update weight w i The expression is:

[0140] w i =(1-α)w i +α

[0141] For mismatched Gaussian distribution, the expression for weight decay is:

[0142] w i =(1-α)w i

[0143] If no Gaussian distribution matches the current pixel value X t , then use X t Create a new Gaussian distribution and delete the Gaussian distribution with the smallest weight;

[0144] S2.3.3. In order to ensure that the sum of all weights is 1, all weights are normalized, and the expression is:

[0145]

[0146] For each pixel, if the current pixel value X t cannot be described by any Gaussian distribution, that is, d i >2.5σ, the pixel is considered to belong to the foreground image.

[0147] S2.4. Train the YOLOv5 model using an image dataset containing various road spills, obtain the trained YOLOv5 model as an AI target detection model, and use the trained AI target detection model to detect the road image obtained in step S2.3 to obtain the output image detected by the video recognition module;

[0148] YOLOv5 (You Only Look Once version 5) is a convolutional neural network architecture for object detection. It adopts a single-stage detector strategy to detect and classify multiple objects simultaneously through a single forward propagation. The following is a detailed description of the YOLOv5 structure:

[0149] Input layer: The input layer receives the image and normalizes it to a fixed size, such as 640x640. The image is preprocessed in the input layer, including scaling, normalization, etc.

[0150] Backbone network: The backbone network is used to extract image features. YOLOv5 usually uses CSPDarknet as the backbone network, which contains multiple residual blocks (ResBlock), each of which has several convolutional layers, batch normalization layers (Batch Normalization) and activation functions (usually Leaky ReLU). The output of the backbone network is feature maps of different scales, which contain semantic information at different levels.

[0151] Feature Pyramid Network (FPN) and Path Aggregation Network (PAN): FPN and PAN structures are used to fuse feature maps of different scales to improve detection results. FPN (Feature Pyramid Network): Constructs a feature map pyramid from bottom to top, passing information from low-level feature maps to high-level feature maps. PAN (Path Aggregation Network): Passes information from top to bottom and combines feature maps of different scales.

[0152] Detection head: The detection head is used to predict the target box and category on each feature map. Each scale feature map generates three outputs through the convolution layer: objectness: predict whether the area contains an object. Bounding box regression: predict the coordinates and size of the target box. Classification: predict the category of the target.

[0153] Output layer: The output layer combines the detection results of each scale to form the final detection output. The output of each scale contains: Objectivity: the confidence score of the object box. Bounding box: the coordinates of the object box (x, y, w, h). Category: the category label of the object. The final output is a list of all detection results, which include the objectivity, bounding box and category information of each object.

[0154] S2.5. The video image of abnormal lane change behavior obtained in step S2.1 is used to generate a trajectory distribution map based on the vehicle trajectory data of the past period of time. The value of each pixel in the trajectory distribution map represents the total duration of the trajectory at that point in unit time. The distribution map is input into the feature extraction network of the multi-layer convolution layer to obtain the vehicle trajectory embedding vector, and then the vehicle trajectory embedding vector is spliced ​​with the video recognition module detection output image obtained in step S2.3 to obtain the video recognition module output image, which is expressed as:

[0155] Input=Concat(Conv(TrajectoryMap), VideoDetector(Image))

[0156] Among them, TrajectoryMap is the trajectory distribution map, Conv is the multi-layer convolution layer, Image is the original image data, VideoDetector is the detection model in the video recognition module, and Input is the output image of the video recognition module.

[0157] S3. The radar recognition module preprocesses the radar image obtained in step S1, then performs image background modeling to obtain a processed radar image, and then uses the trained 3D target detection model to perform target detection and recognition on the processed radar data to obtain an output image of the radar recognition module;

[0158] Furthermore, the specific implementation method of step S3 includes the following steps:

[0159] S3.1. The radar recognition module preprocesses the radar image obtained in step S1 by performing denoising processing using the surface outlier removal SOR method and median filtering to obtain a preprocessed radar image;

[0160] Furthermore, the parameters of the SOR algorithm usually include the standard deviation range (nSigma) and the minimum number of neighborhood points (k). Calculate the average distance: For each point p in the point cloud i , find its k nearest neighbors. These neighbors are denoted as {p i1 ,p i2 ,...,p ik}.

[0161] Calculate point p i To each neighbor p ij The Euclidean distance d ij :

[0162] d ij =∥p i -p ij ∥

[0163] Point p i The average distance for:

[0164]

[0165] Calculate the mean and standard deviation of the distances, and calculate the average distance of all points Afterwards, the overall mean μ and standard deviation σ of these average distances are calculated:

[0166]

[0167] Where N is the total number of points in the point cloud.

[0168] Identify outliers. If a point p i The average distance If the distance is greater than the mean distance plus a certain number of times the standard deviation (threshold), the point is considered an outlier. This threshold is defined as:

[0169] threshold=μ+ασ

[0170] if Then p i are considered as outliers and removed from the point cloud.

[0171] The median filter requires the selection of an appropriate filter window size to eliminate noise based on the characteristics of the data.

[0172] S3.2. Perform background modeling on the preprocessed radar image obtained in step S3.1, first voxelize the radar point cloud data, select a grid size, distribute the points to prime voxels, and then perform historical statistical modeling on the spatial voxels by using a Gaussian mixture model (GMM) model suitable for dynamic backgrounds, adjust the parameters of the GMM model, including the number of Gaussian distributions and the covariance matrix, remove moving targets including vehicles and pedestrians, retain static background information data, and obtain a processed radar image;

[0173] S3.3. Use the trained 3D target detection model to perform target detection and recognition on the processed radar data obtained in step S3.2. Use the PointPillars network to adjust the learning rate, batch size, and number of training rounds to obtain a model with good detection capabilities to detect 3D targets on the ground and obtain the output image of the radar recognition module.

[0174] S4. The video recognition module output image obtained in step S2 and the radar recognition module output image obtained in step S3 are input into the spilled object recognition module for integration, and the suspected spilled objects are judged, identified and classified to obtain an output image for identifying the spilled objects;

[0175] Furthermore, the specific implementation method of step S4 includes the following steps:

[0176] S4.1. Input the output image of the video recognition module obtained in step S2 into the AI ​​classification model in the spilled object recognition module, and use the modified ResNet-50 model as the AI ​​classification model. The ResNet-50 model structure is as follows:

[0177] ResNet-50=Dense(AvgPooling(Conv5_x(Conv4_x(Conv3_x(Conv2_x(Conv1(Input)))))))

[0178] Among them, AvgPooling is the average pooling layer, Conv1, Conv2_x, Conv3_x, Conv4_x, Conv5_x are the convolution modules of the first layer depth, the convolution modules of the second layer depth, the convolution modules of the third layer depth, the convolution modules of the fourth layer depth, and the convolution modules of the fifth layer depth respectively, and Dense is the classification result output by the fully connected layer after the SoftMax activation function;

[0179] The role of the classification network is to conduct secondary confirmation of the detection results and eliminate interference such as light spots and shadows. In this method, a modified ResNet-50 model is used as an AI classification model. The model receives the RGB image detected by Yolov5 and the four-channel image composed of the vehicle trajectory embedding vector as comfort. An image dataset containing various road spills and non-road spills is used to train the ResNet-50 model so that it can distinguish between road spills and interference factors such as light spots and shadows. The classification model will dynamically adjust the possibility of classification according to the value of the vehicle trajectory embedding vector. When there is an obvious distribution of lane change trajectories, the classification model can more easily determine suspected spills as correct spills, and vice versa, it is easier to filter out falsely detected light spots, shadows, etc.

[0180] Conv1:7x7.64filters, stride 2

[0181] 1x1.64 filters

[0182] Conv2_x: [3x3, 64filters]×3

[0183] 1x1, 256 filters

[0184] 1x1, 128 filters

[0185] Conv3_x: [3x3, 128filters]×4

[0186] 1x1, 512 filters

[0187] 1x1, 256 filters

[0188] Conv4_x: [3x3, 256filters]×6

[0189] 1x1, 1024 filters

[0190] 1x1, 512 filters

[0191] Conv5_x: [3x3, 512filters]×3

[0192] 1x1, 2048 filters

[0193] S4.2. Input the radar recognition module output image obtained in step S3 into the deep learning-based point cloud classification model PointNet in the spilled object recognition module, and perform secondary classification on the target suspected to be a spilled object to determine whether the target is a spilled object. The expression of PointNet is:

[0194] PointNet=SoTtMax(MaxPooling(SoftMax(T-Nett(MLP(T-Net(PointCloud))))))

[0195] Among them, T-Net is the input transformation network used to learn the transformation matrix of the input point cloud, MLP is a multi-layer shared multi-layer perceptron, PointCloud is the point cloud data, and MaxPooling is the maximum pooling layer; finally, SoftMax is used to predict the final classification score, and the expression is:

[0196] T-Net(Input)=MLP k×n (Input)×MLP n×n (MLP k×n (Input));

[0197] S4.3. Use the multi-frame image judgment method to make a comprehensive judgment on the video detection result obtained in step S4.1 and the radar classification result obtained in step S4.2. If the video classification results at multiple moments are consistent with the radar detection classification results, the detection result of the spilled object is considered to be credible. If the image classification results at multiple moments are inconsistent with the classification results, the detection result is considered to be unreliable and will be eliminated. If it is determined to be a spilled object, a rectangular frame is used to mark the road spilled object, and an alarm message of abnormal road events is prompted in the system. The information is stored in the database, and the vehicle data before and after the spilled object is retrieved for a period of time, and the vehicle information recognition module of the spilled object is input;

[0198] Furthermore, the trained spill classification model can be used to determine the specific type of the spill; if it is a false detection, it will be ignored and no marking or alarm will be performed.

[0199] S5. The output image obtained in step S4 for identifying the scattered objects is input into the scattered object vehicle information recognition module to extract the video clip, license plate information or RFID information of the scattered object vehicle;

[0200] Furthermore, the specific implementation method of step S5 includes the following steps:

[0201] S5.1. The output image of the scattered object obtained in step S4 is input into the scattered object vehicle information recognition module. When the scattered object recognition module recognizes the scattered object information, firstly, according to the recognition timestamp, the scattered vehicle ID is traced from the historical vehicle identification information according to the location, trajectory and other information, and the video recording is started to intercept the video clips before and after the scattering behavior for a certain time;

[0202] S5.2. Detect and crop the scattered vehicle image from the captured video image using the deep learning-based vehicle detection model Yolov5;

[0203] The backbone network of YOLOv5 adopts a lightweight model architecture. CSPDarknet53 and EfficientNet are selected as the basic networks, which are responsible for extracting features from the input image. The feature vectors are then used for further processing and fusion of feature maps through the Neck structure. Finally, the detection head is responsible for generating prediction boxes at different scales and predicting the category probability and bounding box position of each box.

[0204] Use the license plate detection model based on EfficientDet to detect the scattered vehicle image. If the license plate can be detected, the license plate image is cropped out, and the perspective transformation matrix of the license plate is calculated based on the detected license plate corner data. The license plate image is projected into a front view, and the license plate recognition PaddleOCR model is used to extract the license plate information. If the license plate cannot be detected, the RFID information of the scattered vehicle is recorded.

[0205] Furthermore, the step of projecting the license plate in step S5.2 is to first obtain the corner points of the license plate:

[0206] P={(x1,y1),(x2,y2),(x3,y3),(x4,y4)};

[0207] Then calculate the perspective transformation matrix M, the expression is:

[0208] M=getPerspectiveTransform(P,P′)

[0209] Among them, getPerspectiveTransform is used to calculate the perspective transformation matrix, P is the corner point of the license plate in the image, and P′ is the corner point of the license plate in the plane rectangular coordinate system established according to the actual size;

[0210] The license plate image is transformed into a perspective view to obtain the front view I′, which is expressed as:

[0211] I′=warpPerspective(I,M)

[0212] Among them, I is the actual image of the license plate, and I′ is the front view of the license plate.

[0213] S6. The video clip, license plate information or RFID information of the vehicle that spilled the objects extracted in step S5 is transmitted to the road section company and the road administration law enforcement department, prompting them to handle the dynamic and static spilled objects on the highway.

[0214] What are the key technical points and the points to be protected of the present invention?

[0215] The present invention provides a device, system and method for identifying and detecting dynamic and static spilled objects based on a multimodal pre-trained large model in a highway scenario, including:

[0216] Device: integrated radar camera and edge computing gateway.

[0217] System: video acquisition module, video recognition module, radar recognition module, spilled object recognition module, spilled object vehicle information recognition module.

[0218] Method: The spilled object recognition and judgment model training and target recognition and tracking method are based on multimodal pre-trained large model learning technology.

[0219] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0220] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A system for identifying and detecting dynamic and static spilled objects on a highway, characterized in that: It includes a radar-vision integrated machine, a video recognition module, a radar recognition module, a spilled object recognition module, a spilled object vehicle information recognition module, and a host computer; The integrated radar and vision machine is respectively connected to a video recognition module and a radar recognition module, the video recognition module and the radar recognition module are respectively connected to a spilled object recognition module, the spilled object recognition module is connected to a spilled object vehicle information recognition module, and the spilled object vehicle information recognition module is connected to a host computer; The camera and radar of the integrated radar and vision machine respectively collect real-time images on the highway, and the network interfaces of the camera, radar and edge computing gateway are connected via Ethernet cables; The video recognition module is used to detect images of abnormal objects on the highway and images of abnormal vehicle lane change behavior on the highway; The radar recognition module is used to detect point cloud data of abnormal objects on the highway; The spilled object recognition module is used to integrate the outputs of the video recognition module and the radar recognition module to identify suspected spilled objects; The object-scattering vehicle information identification module is used to identify the video clip, license plate information or RFID information of the object-scattering vehicle; The host computer receives the video clip, license plate information or RFID information of the object-scattering vehicle obtained by the object-scattering vehicle information identification module.

2. A system for identifying and detecting dynamic and static spilled objects on a highway according to claim 1, characterized in that: The edge computing gateway accesses the real-time video stream information collected by the camera of the RayVision integrated device through the RTSP video stream address.

3. A method for identifying and detecting dynamic and static scattered objects on a highway, implemented according to a system for identifying and detecting dynamic and static scattered objects on a highway as claimed in any one of claims 1-2, characterized in that: The steps include: S1. The camera and radar of the integrated machine collect real-time images on the highway and transmit them to the video recognition module and radar recognition module respectively; S2. The video recognition module determines abnormal vehicle lane change behavior on the video image obtained in step S1, then preprocesses the video image obtained in step S1, then performs image background modeling to obtain a road surface image, and then uses the trained AI target detection model to detect the road surface image and identify the output image of the video recognition module; S3. The radar recognition module preprocesses the radar image obtained in step S1, then performs image background modeling to obtain a processed radar image, and then uses the trained 3D target detection model to perform target detection and recognition on the processed radar data to obtain an output image of the radar recognition module; S4. The video recognition module output image obtained in step S2 and the radar recognition module output image obtained in step S3 are input into the spilled object recognition module for integration, and the suspected spilled objects are judged, identified and classified to obtain an output image for identifying the spilled objects; S5. The output image obtained in step S4 for identifying the scattered objects is input into the scattered object vehicle information recognition module to extract the video clip, license plate information or RFID information of the scattered object vehicle; S6. The video clip, license plate information or RFID information of the vehicle that spilled the objects extracted in step S5 is transmitted to the road section company and the road administration law enforcement department, prompting them to handle the dynamic and static spilled objects on the highway.

4. A method for identifying and detecting dynamic and static spilled objects on a highway according to claim 3, characterized in that: The specific implementation method of step S2 includes the following steps: S2.

1. The video recognition module collects the vehicle trajectory data T from the video image obtained in step S1, and the expression is: Where N is the number of data points, (x i ,y i ) is the trajectory coordinate of the i-th data point; The trajectory is smoothed by a Gaussian filter to obtain the smoothed trajectory T′, which is expressed as: T′=GaussiamFilter(T,σ) Where σ is the standard deviation of the Gaussian filter, and then the trajectory is divided into k segments of fixed length L, and the expression is: Then calculate the velocity v on the segmented trajectory i , acceleration a i and the rate of change of the angular direction Δθ i , the expression is: Dth i =θ i+1 -θ i in, is the direction angle. When one or more of the speed, acceleration and direction angle change rate is greater than a threshold, it is considered that an abnormal lane change behavior exists, and a video image of the abnormal lane change behavior is obtained; S2.

2. The video recognition module preprocesses the video image obtained in step S1, including image distortion correction, adaptive illumination enhancement, and expansion of the training sample set, to obtain a preprocessed video image; S2.

3. Perform image background modeling on the preprocessed video image obtained in step S2.2, and use the Gaussian mixture model GMM model suitable for dynamic backgrounds to remove moving targets including vehicles and pedestrians, retain the road surface area as the area to be detected, and obtain a road surface image; S2.

4. Train the YOLOv5 model using an image dataset containing various road spills, obtain the trained YOLOv5 model as an AI target detection model, and use the trained AI target detection model to detect the road image obtained in step S2.3 to obtain the output image detected by the video recognition module; S2.

5. The video image of abnormal lane change behavior obtained in step S2.1 is used to generate a trajectory distribution map based on the vehicle trajectory data of the past period of time. The value of each pixel in the trajectory distribution map represents the total duration of the trajectory at that point in unit time. The distribution map is input into the feature extraction network of the multi-layer convolution layer to obtain the vehicle trajectory embedding vector, and then the vehicle trajectory embedding vector is spliced ​​with the video recognition module detection output image obtained in step S2.3 to obtain the video recognition module output image, which is expressed as: Input=Concat(Conv(TrajectoryMap), VideoDetector(Image)) Among them, TrajectoryMap is the trajectory distribution map, Conv is the multi-layer convolution layer, Image is the original image data, VideoDetector is the detection model in the video recognition module, and Input is the output image of the video recognition module.

5. A method for identifying and detecting dynamic and static spilled objects on a highway according to claim 4, characterized in that: The method for constructing the Gaussian mixture model GMM model suitable for dynamic background in step S2.3 includes the following steps: S2.3.

1. Set the color value of each pixel to be described by multiple Gaussian distributions. Assuming that each pixel value is X, the expression of the Gaussian mixture model P(X) is: Where K is the number of Gaussian distributions, w i is the weight of the ith Gaussian distribution, μ i is the mean vector of the i-th Gaussian distribution, ∑ i is the covariance matrix of the ith Gaussian distribution, The mean is μ i The covariance matrix is ​​Σ i Gaussian distribution of S2.3.

2. For each new pixel value X t , the Gaussian mixture model performs the following update steps: Calculate pixel value X t The distance d from each Gaussian distribution i , the expression is: d i =||X t -m i || Then determine the current pixel value X t Whether it matches a Gaussian distribution, if d i If it is less than a certain threshold, usually 2.5σ, then X is considered t Belongs to the i-th Gaussian distribution; For the matching Gaussian distribution Update the parameters and the expression for updating the mean vector is: m i =(1-a)m i +αX t Update the covariance matrix Σ i The expression is: S i =(1-α)∑ i +α(X t -m i )(X t -m i ) T Update weight w i The expression is: w i =(1-α)w i +a For mismatched Gaussian distribution, the expression for weight decay is: w i =(1-α)w i If no Gaussian distribution matches the current pixel value X t , then use X t Create a new Gaussian distribution and delete the Gaussian distribution with the smallest weight; S2.3.

3. In order to ensure that the sum of all weights is 1, all weights are normalized, and the expression is: For each pixel, if the current pixel value X t cannot be described by any Gaussian distribution, that is, d i >2.5σ, the pixel is considered to belong to the foreground image.

6. A method for identifying and detecting dynamic and static spilled objects on a highway according to claim 5, characterized in that: The specific implementation method of step S3 includes the following steps: S3.

1. The radar recognition module preprocesses the radar image obtained in step S1 by performing denoising processing using the surface outlier removal SOR method and median filtering to obtain a preprocessed radar image; S3.

2. Perform background modeling on the preprocessed radar image obtained in step S3.1, first voxelize the radar point cloud data, select a grid size, distribute the points to prime voxels, and then perform historical statistical modeling on the spatial voxels by using a Gaussian mixture model (GMM) model suitable for dynamic backgrounds, adjust the parameters of the GMM model, including the number of Gaussian distributions and the covariance matrix, remove moving targets including vehicles and pedestrians, retain static background information data, and obtain a processed radar image; S3.

3. Use the trained 3D target detection model to perform target detection and recognition on the processed radar data obtained in step S3.

2. Use the PointPillars network to adjust the learning rate, batch size, and number of training rounds to obtain a model with good detection capabilities to detect 3D targets on the ground and obtain the output image of the radar recognition module.

7. A method for identifying and detecting dynamic and static scattered objects on a highway according to claim 6, characterized in that: The specific implementation method of step S4 includes the following steps: S4.

1. Input the output image of the video recognition module obtained in step S2 into the AI ​​classification model in the spilled object recognition module, and use the modified ResNet-50 model as the AI ​​classification model. The ResNet-50 model structure is as follows: ResNet-50 =Dense(AvgPooling(Conv5_x(Conv4_x(Conv3_x(Conv2_x(Conv1(Imput))))))) Among them, AvgPooling is the average pooling layer, Conv1, Conv2_x, Conv3_x, Conv4_x, Conv5_x are the convolution modules of the first layer depth, the convolution modules of the second layer depth, the convolution modules of the third layer depth, the convolution modules of the fourth layer depth, and the convolution modules of the fifth layer depth respectively, and Dense is the classification result output by the fully connected layer after the SoftMax activation function; S4.

2. Input the radar recognition module output image obtained in step S3 into the deep learning-based point cloud classification model PointNet in the spilled object recognition module, and perform secondary classification on the target suspected to be a spilled object to determine whether the target is a spilled object. The expression of PointNet is: PointNet=SoftMax(MaxPooling(SoftMax(T-Nett(MLP(T-Net(PointCloud)))))) Among them, T-Net is the input transformation network used to learn the transformation matrix of the input point cloud, MLP is a multi-layer shared multi-layer perceptron, PointCloud is the point cloud data, and MaxPooling is the maximum pooling layer; finally, SoftMax is used to predict the final classification score, and the expression is: T-Net(Input)=MLP k×n (Input)×MLP n×n (MLP k×n (Input)); S4.

3. Use the multi-frame image judgment method to make a comprehensive judgment on the video detection results obtained in step S4.1 and the radar classification results obtained in step S4.

2. If the video classification results at multiple moments are consistent with the radar detection classification results, the spilled object detection results are considered credible. If the image classification results at multiple moments are inconsistent with the classification results, the detection results are considered unreliable and will be eliminated. If it is determined to be spilled objects, a rectangular frame will be used to mark the spilled objects on the road surface, and an abnormal road event alarm message will be prompted in the system. The information will be stored in the database, and the vehicle data for a period of time before and after the spilled objects will be retrieved and input into the spilled object vehicle information recognition module.

8. A method for identifying and detecting dynamic and static spilled objects on a highway according to claim 7, characterized in that: The specific implementation method of step S5 includes the following steps: S5.

1. The output image of the scattered object obtained in step S4 is input into the scattered object vehicle information recognition module. When the scattered object recognition module recognizes the scattered object information, firstly, according to the recognition timestamp, the scattered vehicle ID is traced from the historical vehicle identification information according to the location, trajectory and other information, and the video recording is started to intercept the video clips before and after the scattering behavior for a certain time; S5.

2. Detect and crop the scattered vehicle image from the captured video image using the deep learning-based vehicle detection model Yolov5; The backbone network of YOLOv5 adopts a lightweight model architecture. CSPDarknet53 and EfficientNet are selected as the basic networks, which are responsible for extracting features from the input image. The feature vectors are then used for further processing and fusion of feature maps through the Neck structure. Finally, the detection head is responsible for generating prediction boxes at different scales and predicting the category probability and bounding box position of each box. The scattered vehicle images are detected using the EfficientDet-based license plate detection model. If the license plate can be detected, the license plate image is cropped out, the perspective transformation matrix of the license plate is calculated based on the detected license plate corner point data, and the license plate image is projected into a front view. The license plate information is extracted using the license plate recognition PaddleOCR model; if the license plate cannot be detected, the RFID information of the scattered vehicle is recorded.

9. A method for identifying and detecting dynamic and static spilled objects on a highway according to claim 8, characterized in that: Step S5.2 The steps of projecting the license plate are as follows: first, obtain the corner points of the license plate P = {(x1, y1), (x2, y2), (x3, y3), (x4, y4)}; Then calculate the perspective transformation matrix M, the expression is: M=getPerspectiveTransform(P,P′) Among them, getPerspectiveTransform is used to calculate the perspective transformation matrix, P is the corner point of the license plate in the image, and P′ is the corner point of the license plate in the plane rectangular coordinate system established according to the actual size; The license plate image is transformed into a perspective view to obtain the front view I′, which is expressed as: I′=warpPerspective(I,M,(w p ,h p )) Among them, I is the actual image of the license plate, and I′ is the front view of the license plate.

Images