On-road parking automatic identification evidence obtaining and ETC payment all-in-one machine

Abstract

The invention relates to the technical field of Internet of Things intelligent equipment, in particular to an on-road parking automatic identification evidence obtaining and ETC payment all-in-one machine. The system comprises a parking space monitoring module, a model construction module, a data processing module and an ETC payment module. According to the invention, the parking space is monitored through the parking space monitoring module, the image is preprocessed through the image processing function of OpenCV, the processed image is transmitted to the trained YOLO convolutional neural network model in the model construction module, the state of the vehicle in the parking space is dynamically tracked, unmanned monitoring of the vehicle entering the parking space is realized, and the parking space monitoring efficiency is improved. When the vehicle is parked in the parking space, the data processing module records the driving-in time of the vehicle, and starts to calculate the parking fee and generate a corresponding bill according to a preset charging rule, and the ETC payment module carries out fee deduction, thereby achieving the unmanned fee deduction of the parking space, and avoiding the fee evasion phenomenon.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things intelligent devices, and in particular to an integrated machine for automatic identification and evidence collection of on-street parking and ETC payment. Background Art

[0002] Automatic identification and evidence collection of on-street parking is a measure that uses advanced technical means to improve the efficiency of urban parking management and solve the problem of parking difficulties. The system uses a variety of technical means, such as video surveillance, geomagnetic induction, and intelligent patrol vehicles, to achieve automatic identification, evidence collection, and management of on-street parking behaviors. The management of public parking spaces on both sides of urban municipal roads mainly relies on manual recording and manual charging, and auxiliary technical means include ground magnetic induction and visual recognition technologies, which have achieved a certain degree of semi-automation in the evidence collection of parking behaviors.

[0003] Since on-street parking is an open parking scene, although there are mobile payment channels, it is often necessary for people to go to the site to cooperate with the collection of fees. There is a relatively large proportion of unpaid fees and evasion of fees. In addition, when the parking spaces are unmanned, due to weather reasons and interference objects attached near the vehicle's license plate, the high-definition camera cannot accurately identify the vehicle's license plate, and thus cannot complete the subsequent deduction operation. Therefore, we propose an on-street parking automatic identification and evidence collection and ETC payment all-in-one machine. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that when parking spaces are managed without human monitoring, due to weather reasons and interference objects attached near the vehicle's license plate, the high-definition camera cannot accurately identify the vehicle's license plate, and thus cannot complete the subsequent fee deduction operation. In order to be able to more accurately identify the vehicle and record the vehicle's parking time, fully automatic monitoring and charging of public parking spaces are realized.

[0005] To achieve the above-mentioned purpose, the present invention provides an integrated machine for automatic identification and evidence collection of on-street parking and ETC payment, including a parking space monitoring module, a model building module, a data processing module and an ETC payment module;

[0006] The parking space monitoring module monitors the parking space through a camera with an 8-megapixel high-definition video stream, and uses the background difference method to perform a difference operation between the current frame image and the background image to determine whether the vehicle begins to enter the parking space. When it is determined that the vehicle begins to enter the parking space, the image is filtered and enhanced through the image processing function of OpenCV, and the image is pre-processed, and the pre-processed image is passed to the model building module;

[0007] The model building module uses the video stream monitored by the parking space monitoring module as the input layer, and the bounding box, confidence and category probability of the vehicle as the output layer to build a YOLO convolutional neural network model, uses a professional image annotation tool LabelImg to extract the vehicles in the image frame by frame in the video stream and annotate them, uses a mean square error loss function to measure the difference between the predicted result and the actual annotation, trains the YOLO convolutional neural network model, inputs the image preprocessed by the parking space monitoring module into the YOLO convolutional neural network model, determines whether the vehicle is parked in the parking space, and then extracts the license plate, front, rear and wheel features of the vehicle, and transmits the feature data of the vehicle to the data processing module;

[0008] The data processing module receives the vehicle feature data transmitted by the model building module and stores it. At the same time, it records the vehicle's entry time and starts to calculate the parking fee according to the preset charging rules. When the vehicle is about to leave the parking space, the parking space monitoring module collects the vehicle image again, extracts the vehicle's feature data through the model building module, and compares the data with the vehicle feature data stored when parking to determine whether the vehicle is the same vehicle. When the four wheels of the vehicle are not in the parking space, the charging is stopped, and the corresponding bill is generated according to the charging rules, and the bill is transmitted to the ETC payment module;

[0009] The ETC payment module detects that the parked vehicle leaves the parking space, and automatically triggers the ETC antenna to open without delay within the same device system to perform a radio frequency scan of the target area within a certain range near the parking space. A trusted OBU is detected and compared with the current vehicle license plate. After the comparison is consistent, a deduction is initiated. After the deduction is successful, the national or provincial ETC center sends a text message to notify the car owner about the parking consumption.

[0010] As a further improvement of the technical solution, the model building module includes a training model unit and a feature extraction unit;

[0011] The training model unit labels vehicles in an image data set collected from vehicles parked in parking spaces, divides the data set into a training set, a validation set, and a test set in proportion, trains the model with a minimum mean square error loss function as a target, and adaptively adjusts the learning rate through an Adam optimizer to make the training process more stable and efficient;

[0012] The feature extraction unit uses a camera with an 8-megapixel high-definition video stream to take photos of the license plate, front, rear and wheels of the vehicle parked in the parking space, and extracts the features of the license plate, front, rear and wheels of the vehicle.

[0013] As a further improvement of the present technical solution, the training model unit performs data enhancement operations on the image when training the model, flips the image horizontally and vertically, adds Gaussian noise, and then performs training again.

[0014] As a further improvement of the present technical solution, the training model unit removes redundant bounding boxes through a non-maximum suppression algorithm, screens out the bounding box that is closest to the real car bounding box and has the highest confidence, and removes other bounding boxes that overlap more and are relatively inaccurate, thereby obtaining clear and accurate car target detection results.

[0015] As a further improvement of the technical solution, the feature extraction unit includes a license plate recognition component and a shape recognition component;

[0016] The license plate recognition component locates the position of the license plate through color space conversion and threshold segmentation, and then uses the edge detection algorithm to obtain the edge information of the license plate to accurately locate the license plate;

[0017] The shape recognition component extracts the features of the front, rear and wheels of the vehicle through a texture recognition method and a Hough circle transformation method.

[0018] As a further improvement of the technical solution, after locating the license plate, the license plate recognition component uses a character segmentation and recognition algorithm to recognize the license plate number.

[0019] As a further improvement of the technical solution, the parking space detection module includes a parking determination unit and a preprocessing unit;

[0020] The parking determination unit compares the vehicle area in the differential image with the parking space area according to the obtained differential image, and if the extracted vehicle area position overlaps with the parking space area, it can be preliminarily determined that the vehicle begins to enter the parking space;

[0021] When the preprocessing unit preprocesses the image, a median filtering method is used to replace the central pixel with the median of the neighboring pixels to remove salt and pepper noise.

[0022] As a further improvement of the present technical solution, the parking space monitoring module is provided with a parking time judgment mechanism. When the four wheels of the vehicle are all inside the parking space, a certain number of frames are continuously observed. If the wheel position of the vehicle does not change significantly during this period of time, it is finally determined that the vehicle has entered the parking space.

[0023] As a further improvement of the present technical solution, when the data processing module compares the extracted vehicle feature data with the vehicle feature data stored when parking, it uses a weighted fusion method to weightedly fuse the license plate features and vehicle appearance features to determine whether the vehicles are the same.

[0024] As a further improvement of the present technical solution, the ETC payment module adopts multi-threaded communication, utilizes an event-driven architecture to handle various events in the system, and uses a message queue to transmit events and data.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. The automatic identification and evidence collection and ETC payment integrated machine for on-street parking monitors the parking space through the parking space monitoring module. When the vehicle starts to enter the parking space, the video stream at this time is intercepted, and the image is pre-processed through the image processing function of OpenCV. The processed image is passed to the trained YOLO convolutional neural network model in the model building module to obtain the bounding box, confidence and category probability of the vehicle, dynamically track the status of the vehicle in the parking space, and extract the license plate, front, rear and wheel features of the vehicle, so as to realize unmanned monitoring of the vehicle entering the parking space and automatically determine whether the vehicle is parked in the parking space;

[0027] 2. When a vehicle is parked in a parking space, the data processing module records the vehicle's entry time and starts calculating the parking fee according to the preset charging rules. When the vehicle is about to leave the parking space, the vehicle feature data is extracted again and compared with the vehicle feature data stored when parking, and the vehicle is verified twice to ensure the accuracy of vehicle identification. When the four wheels of the vehicle are not in the parking space, the charging is stopped, and the corresponding bill is generated according to the charging rules. The ETC payment module deducts the fee, realizing unmanned deduction of parking fees and avoiding fee evasion;

[0028] 3. The on-street parking automatic identification and evidence collection and ETC payment integrated machine adopts multi-threaded communication and asynchronous processing mechanism, placing the vehicle detection task in an independent thread, which can avoid the task from occupying too many system resources, allowing other tasks to be carried out simultaneously, improving the system's parallel processing capabilities and improving the system's response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0030] Figure 2 It is a schematic diagram of the overall details of the present invention;

[0031] Figure 3 It is a schematic diagram of the feature extraction unit flow of the present invention.

[0032] The meaning of each number in the figure is:

[0033] 100. Parking space monitoring module; 110. Parking determination unit; 120. Preprocessing unit; 200. Model building module; 210. Training model unit; 220. Feature extraction unit; 221. License plate recognition component; 222. Shape recognition component; 300. Data processing module; 400. ETC payment module. DETAILED DESCRIPTION

[0034] The following will be combined with the accompanying drawings in the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] At present, the management of public parking spaces on both sides of urban municipal roads mainly relies on manual recording and manual charging, and there are problems such as high construction and operation costs, large labor input, and high arrears rate. In order to concentrate the processes of road parking evidence collection, billing, and ETC contactless payment on one device, and at the same time, solve the problem that when unmanned monitoring and management of parking spaces is carried out, due to weather reasons and interference objects attached near the vehicle's license plate, the high-definition camera cannot accurately identify the vehicle's license plate, and thus cannot complete the subsequent deduction operation. In order to be able to more accurately identify vehicles, record the parking time of vehicles, and realize fully automatic monitoring and charging of public parking spaces.

[0036] Therefore, the present invention proposes to monitor the parking space through the parking space monitoring module, transmit the monitored image to the YOLO convolutional neural network model trained in the model construction module, dynamically track the status of the vehicle in the parking space, record the vehicle's entry time through the data processing module, and start calculating the parking fee according to the preset billing rules, and then the ETC payment module deducts the fee to realize unmanned deduction of the parking space.

[0037] The details are as follows:

[0038] See also Figure 1 As shown, the present invention provides an integrated machine for automatic identification and evidence collection of on-street parking and ETC payment, including a parking space monitoring module 100, a model building module 200, a data processing module 300 and an ETC payment module 400;

[0039] The parking space monitoring module 100 monitors the parking space through a camera with an 8-megapixel high-definition video stream, and uses a background difference method to perform a difference operation between the current frame image and the background image to determine whether the vehicle begins to enter the parking space. When it is determined that the vehicle begins to enter the parking space, the image is filtered and enhanced through the image processing function of OpenCV, and the image is pre-processed, and the pre-processed image is passed to the model building module 200;

[0040] The model building module 200 uses the video stream monitored by the parking space monitoring module 100 as the input layer, and the bounding box, confidence and category probability of the vehicle as the output layer to build a YOLO convolutional neural network model, and uses the professional image annotation tool LabelImg to annotate the vehicles in the image extracted frame by frame in the video stream, and uses the mean square error loss function to measure the difference between the predicted result and the actual annotation, and trains the YOLO convolutional neural network model. The image pre-processed by the parking space monitoring module 100 is input into the YOLO convolutional neural network model to determine whether the vehicle is parked in the parking space, and then extract the license plate, front, rear and wheel features of the vehicle, and transmit the feature data of the vehicle to the data processing module 300;

[0041] The data processing module 300 receives the vehicle feature data transmitted by the model building module 200, stores it, records the vehicle's entry time, and starts calculating the parking fee according to the preset charging rules. When the vehicle is about to leave the parking space, the parking space monitoring module 100 collects the vehicle image again, extracts the vehicle feature data through the model building module 200, and compares the data with the vehicle feature data stored when parking to determine whether the vehicle is the same vehicle. When the four wheels of the vehicle are not in the parking space, the charging is stopped, and the corresponding bill is generated according to the charging rules, and the bill is transmitted to the ETC payment module 400;

[0042] The ETC payment module 400 detects that the parked vehicle leaves the parking space, and automatically triggers the ETC antenna to open without delay within the same device system to perform a radio frequency scan of the target area within a certain range near the parking space. A trusted OBU is detected and compared with the license plate of the current vehicle. After the comparison is consistent, a deduction is initiated. After the deduction is successful, the national or provincial ETC center sends a text message to the car owner to notify him of the parking consumption.

[0043] like Figure 2 As shown, the model building module 200 includes a training model unit 210 and a feature extraction unit 220;

[0044] The training model unit 210 labels the vehicles in the image data set collected from the parking space, divides the data set into a training set, a validation set, and a test set according to the ratio, trains the model with the minimum mean square error loss function (MSE) as the target, and adaptively adjusts the learning rate through the Adam optimizer to make the training process more stable and efficient;

[0045] The feature extraction unit 220 uses a camera with an 8-megapixel high-definition video stream to take a picture of the license plate, front, rear, and wheels of the vehicle parked in the parking space, and extracts the features of the license plate, front, rear, and wheels of the vehicle;

[0046] Among them, the mean square error loss function includes bounding box position prediction, target confidence prediction and target category probability prediction, as follows:

[0047] Bounding box position loss: For the target positioning task, the YOLO model predicts the center coordinates of the bounding box, as well as the width and height of the bounding box. When calculating the position loss, the prediction error of the coordinates and size is measured using MSE. For example, suppose the predicted center coordinates of the bounding box are The true center coordinates are (x, y), then this part of the position loss is For width and height, due to their value range and distribution characteristics, they are usually transformed by square root and then MSE is used to calculate the loss to better balance the loss contribution of objects of different sizes.

[0048] Target confidence loss: Each bounding box has a confidence score, which is used to indicate the probability that the bounding box contains the target object and the accuracy of the prediction. MSE is also used when calculating the confidence loss. Suppose the predicted confidence is The true confidence is C, then the confidence loss is

[0049] Category probability loss: MSE can also be used when predicting the category probability of the target object. Assume that the predicted category probability distribution is (For n categories, is a vector of length n, representing the predicted probability of each category), the true category probability distribution is p, then the category probability loss is

[0050] MSE is a simple and intuitive loss function that calculates the mean square error between the predicted value and the true value, which makes the gradient calculation relatively simple and direct during the model training process, easy to understand and implement. MSE is sensitive to outliers. In YOLO, this sensitivity can, to a certain extent, prompt the model to pay more attention to the accurate prediction of key factors such as the bounding box position, because a large error in the bounding box position will lead to a large MSE loss, thereby guiding the model to adjust the parameters more finely to reduce this error. At the same time, since YOLO is multi-task learning, combined with the loss of other tasks, this sensitivity will not be too extreme, and can balance the optimization of different tasks to a certain extent.

[0051] In order to increase the diversity of data and improve the generalization ability of the model, the training model unit 210 performs data enhancement operations on the image when training the model, flips the image horizontally and vertically, adds Gaussian noise, and then performs training again;

[0052] When enhancing the images collected in the dataset, Python is used to flip the images horizontally, vertically, and add Gaussian noise, and the steps for model training are as follows:

[0053] First, use the ImageFolder class in torchvision.datasets to load images and import the image dataset into the database. Then, define the dataset path and load the dataset. Use transforms.ToTensor() to convert the image to tensor format, which is a common format for processing images in PyTorch. Create data loaders for the training set, validation set, and test set respectively to load image data in batches. The data loader can control parameters such as batch size and whether to shuffle the data.

[0054] Use the RandomHorizontalFlip and RandomVerticalFlip classes in torchvision.transforms to implement horizontal and vertical flips. These operations are applied randomly. The probability can be specified when initializing the class. The default is 0.5. Define a custom class to add Gaussian noise. This class needs to inherit from torchvision.transforms.Transform and implement the __call__ method. Then add this custom transformation to the previous transformation combination. Adjust the mean and standard deviation parameters of the Gaussian noise as needed. For example, if you want less noise, you can set the standard deviation to around 0.1.

[0055] In each training cycle (epoch), the model processes batches of images after data augmentation, calculates losses and updates model parameters. After multiple cycles of training, the model is expected to learn more robust features, so that it can better perform classification or other tasks when faced with different transformed images (such as horizontal flipping, vertical flipping and images with Gaussian noise);

[0056] In object recognition tasks, an object may be placed upright in the image, or it may be flipped horizontally or vertically. By flipping the training images horizontally and vertically, the model can learn the features of objects in different directions. In this way, when the model encounters images of the same object in different directions in actual applications, it can still accurately identify them. This improves the generalization ability of the model based on the diversity of the data set and ensures the recognition accuracy of the model.

[0057] In order to better remove redundant bounding boxes, the training model unit 210 removes redundant bounding boxes through a non-maximum suppression algorithm, selects the bounding box that is closest to the real car bounding box and has the highest confidence, and removes other bounding boxes that overlap more and are relatively inaccurate, thereby obtaining a clear and accurate car target detection result;

[0058] The principle of the non-maximum suppression algorithm is: for a real target, in the predicted bounding box, the bounding box with the highest confidence is likely to be the most accurate box that frames the target. Other bounding boxes that overlap more with this high-confidence bounding box are likely to be redundant. For example, in car target detection, if a bounding box has a confidence of 0.9 and another bounding box that overlaps with it a lot and has a confidence of 0.6, then the latter is likely to be an inaccurate prediction and needs to be removed.

[0059] For two bounding boxes A and B, the calculation formula of the bounding box intersection over union (IoU) is:

[0060]

[0061] Among them, Ar(A∩B) is the area of ​​the overlapping part of the two bounding boxes, A r (A∪B) is the total area of ​​the two bounding boxes after merging.

[0062] The calculation steps are as follows:

[0063] Assume that the bounding box is represented by (x1, y1, x2, y2), where (x1, y1) is the coordinate of the upper left corner and (x2, y2) is the coordinate of the lower right corner. For two bounding boxes A = (x 1A ,y 1A ,x 2A ,y 2A ) and B=(x 1B ,y 1B ,x 2B ,y 2B ), first calculate the coordinates of their overlapping parts, the coordinates of the upper left corner of the overlapping part are (max(x 1A ,x 1B ),max(y 1A ,y 1B )), the coordinate of the lower right corner is (min(x 2A ,x 2B ),min(y 2A ,y 2B )), if any value of the lower right corner coordinate of the overlapping part is less than the value of the upper left corner coordinate, it means that the two bounding boxes do not overlap, and IoU = 0, otherwise, calculate the area of ​​the overlapping part:

[0064] A r(A∩B)=(min(x 2A ,x 2B )-max(x 1A ,x 1B ))*(min(y 2A ,y 2B ))

[0065] Then calculate the total area of ​​the two bounding boxes:

[0066] A r (A∪B)=A r (A)+A r (B)-A r (A∩B)

[0067] Finally, IoU is calculated according to the formula.

[0068] Sort all predicted car bounding boxes from high to low according to confidence. For example, suppose there are 5 bounding boxes with confidences of 0.9, 0.8, 0.7, 0.6 and 0.5 respectively. First process the bounding box with a confidence of 0.9, select the first bounding box after sorting, retain it as part of the final result, and mark it as processed. Calculate the IoU between this retained bounding box and other unprocessed bounding boxes. If it is greater than 0.5, it is considered redundant and removed. Continue to select the bounding box with the highest confidence from the remaining unprocessed bounding boxes, repeat the process of calculating IoU and screening until all bounding boxes are processed, thereby obtaining clear and accurate car target detection results.

[0069] like Figure 3 As shown, the feature extraction unit 220 includes a license plate recognition component 221 and a shape recognition component 222;

[0070] The license plate recognition component 221 locates the position of the license plate through color space conversion and threshold segmentation, and then uses the edge detection algorithm to obtain the edge information of the license plate to accurately locate the license plate;

[0071] The shape recognition component 222 extracts the features of the front, rear and wheels of the vehicle by using a texture recognition method and a Hough circle transform method;

[0072] Among them, common color spaces include RGB, HSV, YCrCb, etc. In license plate positioning, HSV and YCrCb color spaces are more advantageous than RGB because license plate colors have more obvious distribution characteristics in these spaces. For example, in the HSV space, the hue (H) value range of blue license plates is relatively fixed, and the hue value of yellow license plates also has a specific range. Threshold segmentation is performed according to the range of license plate colors in a specific color space. For blue license plates, in the HSV space, the hue (H) may be around 100-140, and the saturation (S) and brightness (V) also have corresponding appropriate ranges. A threshold range can be set to extract the area that may be the license plate color. For yellow license plates, similar operations can be performed, except that the hue range is different. In the obtained mask image, the white area represents the part that may be the license plate color, and the black part represents other areas. In order to comprehensively consider blue and yellow license plates (different countries or regions may have different license plates), the two color masks can be logically or operated to expand the possible license plate area range. After threshold segmentation, the possible license plate area obtained may have noise or discontinuous parts. Morphological operations can be used to improve it, such as erosion and dilation. Erosion can remove small noise points, and dilation can fill small holes and make the license plate area more continuous;

[0073] The license plate is initially located by finding the outline of the white area in the mask image, finding the outline, and then filtering out the outline that may be the license plate based on the area, aspect ratio and other features of the outline. The boundary of the license plate can be further accurately adjusted by combining the possible license plate outline and edge detection results obtained previously. The four corner points of the license plate can be refined by finding the intersection of the edge and the outline, or by using edge information. For example, for a license plate outline that has been initially located, strong edges obtained by Canny edge detection are found near it, and the boundary position of the license plate is corrected based on these edges to make the license plate more accurately located. At the same time, methods such as Hough transform can be used to detect straight lines in the edge of the license plate, because the license plate has four edges, and the accurate shape and position of the license plate can be better determined by detecting straight lines. For example, for the detected straight lines, their intersection points can be calculated to determine the four corner points of the license plate. The final position of the license plate is comprehensively determined based on the detected straight lines and the previous contour information, thereby achieving accurate positioning of the license plate.

[0074] In order to more accurately identify the license plate number, the license plate recognition component 221 uses a character segmentation and recognition algorithm to identify the license plate number after locating the license plate;

[0075] Among them, in the located license plate area, after converting the image into a grayscale image, the number of black (or white, depending on the color relationship between the license plate characters and the background) pixels in each column of pixels is calculated to obtain a vertical projection histogram. There are usually gaps between license plate characters, which will appear as valleys on the histogram. For example, assuming that the license plate width is 300 pixels, the number of black pixels in each column is counted from left to right. This number will drop significantly at the character intervals. Similarly, each row of pixels is counted to obtain a horizontal projection histogram, which is used to determine the upper and lower boundaries of the characters. The results of vertical and horizontal projections can roughly divide the area where each character is located;

[0076] Establish a template library containing all possible characters (such as numbers 0-9, letters AZ, etc.). For each segmented character, match it with the characters in the template library one by one. The matching process can be completed by calculating the similarity, for example, calculating their correlation coefficient or the sum of squared differences. The template character with the highest similarity is the recognition result, and the license plate number is simply and directly recognized.

[0077] The parking space detection module 100 includes a parking determination unit 110 and a pre-processing unit 120;

[0078] The parking determination unit 110 compares the vehicle region in the differential image with the parking space region based on the obtained differential image. If the extracted vehicle region position overlaps with the parking space region, it can be preliminarily determined that the vehicle begins to enter the parking space.

[0079] When preprocessing the image, the preprocessing unit 120 uses a median filtering method to replace the central pixel with the median of the neighboring pixels to remove salt and pepper noise;

[0080] The steps for comparing the vehicle area and the parking space area are as follows:

[0081] First, background modeling is required. Methods such as the mean method, median method, and Gaussian mixture model (GMM) can be used to construct a background image. For example, when using the mean method, in a continuous sequence of video frames, multiple frames of images are collected (assuming that 50 frames are collected), and the pixel values ​​of the corresponding pixel positions are averaged to obtain the background image. Then, the current frame image is differentially calculated from the background image to obtain a differential image. For each pixel in the image, the grayscale value of the pixel in the current frame is subtracted from the grayscale value of the corresponding pixel in the background image (assuming that the current frame image is I(x,y), the background image is B(x,y), and the differential image D(x,y) = I(x,y)-B(x,y)). In this way, in the differential image, the pixel values ​​in the area where moving objects such as vehicles are located will be significantly different from those in the background area, and the vehicle area often presents non-zero (or meets a certain threshold range) pixel values;

[0082] The difference image is binarized by setting a suitable threshold. For example, it is found through experiments that when the threshold is set to 30 (this threshold can be adjusted according to the actual scene, image quality and other factors), the vehicle and background noise can be distinguished well. The pixel points with a pixel value greater than 30 in the difference image are set to white (value 255, indicating the candidate point of the vehicle area), and those less than or equal to 30 are set to black (value 0, indicating the background area), and a binary difference image is obtained;

[0083] If it is a parking lot with fixed parking spaces, such as a shopping mall parking lot or a residential parking lot, the staff can manually mark the polygonal or rectangular area of ​​each parking space on a clear bird's-eye view image of the parking lot through image annotation tools (some intelligent monitoring systems have built-in annotation functions, or you can use open source image annotation software such as LabelImg, etc.), and record the coordinate information corresponding to each parking space area;

[0084] After obtaining the vehicle area and parking area, it is necessary to detect whether there is overlap between them. For a simple rectangular area (assuming that the vehicle area is a rectangle Rv(x1v, y1v, x2v, y2v), and the parking area is a rectangle Rp(x1p, y1p, x2p, y2p)), the overlap can be determined by judging whether the two rectangles intersect in the horizontal and vertical directions. That is, when the conditions (x1v<=x2p&&x2v>=x1p)&&(y1v<=y2p&&y2v>=y1p) are met, it means there is overlap;

[0085] In order to better determine whether a vehicle has entered a parking space, the parking space monitoring module 100 is provided with a parking time determination mechanism. When the four wheels of the vehicle are all inside the parking space, a certain number of frames are continuously observed. If the wheel position of the vehicle does not change significantly during this period, the vehicle is finally determined to have entered the parking space.

[0086] Detecting overlap only once may not be sufficient to determine that the vehicle has started to enter the parking space, because the vehicle may briefly pass by the parking space. In this case, a time window (for example, overlap is detected within 3-5 seconds in a row) or a frame range (overlap is detected for 10-15 frames in a row) can be set, and the position change trend of the vehicle area in the parking space area (for example, the center coordinates of the vehicle area gradually move toward the inside of the parking space) can be combined to comprehensively determine that the vehicle has indeed started to enter the parking space, rather than an accidental overlap.

[0087] In order to more accurately determine whether the vehicle is the same vehicle before and after parking in the parking space, the data processing module 300 compares the extracted vehicle feature data with the vehicle feature data stored when parking, and uses a weighted fusion method to perform weighted fusion on the license plate features and the vehicle appearance features to determine whether the vehicles are the same;

[0088] The license plate is an important identification of the vehicle and has a high uniqueness. Under normal circumstances, the license plate feature should be given a higher weight. Generally speaking, the weight of the license plate feature can be set between 0.6-0.8. For example, in a high-precision vehicle recognition system, since the license plate recognition technology is relatively mature and has a high accuracy, the license plate feature weight can be set to 0.7. The vehicle appearance features include the color, model, and body contour of the vehicle. These features also have a certain degree of recognition, but compared to the license plate, their accuracy may be affected by factors such as vehicle modification, lighting conditions, and shooting angles. The weight of the vehicle appearance feature can be set between 0.2-0.4. For example, in an identification scenario that has certain requirements for the overall characteristics of the vehicle, the weight of the vehicle appearance feature is set to 0.3.

[0089] If the similarity score after weighted fusion is greater than or equal to the set threshold, the vehicles can be determined to be the same; if it is less than the threshold, the vehicles are determined to be different.

[0090] In order to achieve that the time from the vehicle leaving the parking space to the completion of the deduction is no more than 1 second, the ETC payment module 400 adopts multi-threaded communication, uses an event-driven architecture to handle various events in the system, and uses a message queue to transmit events and data;

[0091] Multithreading means running multiple threads simultaneously in a program. Each thread can independently perform different tasks. They share resources such as the program's memory space. Multithreaded communication is a mechanism for information exchange between these different threads. For example, after one thread completes a task, it notifies another thread to start executing the corresponding operation.

[0092] Event-driven architecture, in this architecture, the operation of the system is triggered by events. Events can be user operations (such as clicking a button), input from external devices (such as data from a sensor), or some state changes generated within the system. When an event occurs, the corresponding processing function or code block will be called and executed. After processing the event, the system continues to wait for the next event to be triggered. The entire program flow revolves around the generation and processing of events.

[0093] A message queue is a data structure used to transfer messages (including event information and related data) between different components (here, different threads). It follows the first-in-first-out (FIFO) principle. The sender puts the message into one end of the queue, and the receiver takes the message out of the other end of the queue in order for processing. It can effectively decouple the direct dependencies between different threads and improve the scalability and stability of the system.

[0094] By processing different events in parallel through multiple threads, it can respond quickly to various operations and inputs, and the entire system will not be blocked by a time-consuming task. For example, in a graphical interface application, users can respond immediately to operations such as clicking buttons. The message queue decouples the generation and processing of events, and different threads can be developed and expanded independently. New event types and corresponding processing logic can be easily added to the system as long as the communication specifications of the message queue are followed. If a problem occurs in a thread (for example, the thread processing the event encounters an exception), it will not directly affect the normal operation of other threads, the generation of events, and the messages stored in the message queue, which facilitates troubleshooting and recovery.

[0095] In summary, the working principle of this solution is as follows:

[0096] The on-street parking automatic identification and evidence collection and ETC payment integrated machine monitors the parking space through the parking space monitoring module 100. When the vehicle starts to enter the parking space, the video stream at this time is intercepted, and the image is pre-processed through the image processing function of OpenCV, and the processed image is passed to the trained YOLO convolutional neural network model in the model building module 200 to obtain the vehicle's boundary box, confidence and category probability, dynamically track the state of the vehicle in the parking space, and extract the vehicle's license plate, front, rear and wheel features to achieve unmanned monitoring of the vehicle entering the parking space, and automatically determine whether the vehicle is parked in the parking space. When the vehicle is parked in the parking space, the data processing module 300 records the vehicle's entry time and calculates the vehicle's entry time according to the preset time. The parking fee is calculated according to the charging rules. When the vehicle is about to leave the parking space, the vehicle feature data is extracted again and compared with the vehicle feature data stored when parking, and the vehicle is verified twice to ensure the accuracy of vehicle identification. When the four wheels of the vehicle are not in the parking space, the charging is stopped, and the corresponding bill is generated according to the charging rules. The ETC payment module 400 deducts the fee to realize unmanned deduction of parking fees and avoid the phenomenon of fee evasion. The on-street parking automatic identification and evidence collection and ETC payment integrated machine adopts multi-threaded communication and asynchronous processing mechanism, and puts the vehicle detection task in an independent thread, which can avoid the task from occupying too many system resources, so that other tasks can be carried out simultaneously, improve the parallel processing capability of the system, and improve the response speed of the system.

[0097] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An integrated machine for automatic identification and evidence collection of on-street parking and ETC payment, characterized by: It includes a parking space monitoring module (100), a model building module (200), a data processing module (300) and an ETC payment module (400); The parking space monitoring module (100) monitors the parking space through a camera with an 8-megapixel high-definition video stream, performs a differential operation on a current frame image and a background image using a background difference method, and determines whether a vehicle begins to enter the parking space. When it is determined that the vehicle begins to enter the parking space, the image is filtered and enhanced through the image processing function of OpenCV, and the image is pre-processed, and the pre-processed image is transmitted to the model building module (200); The model building module (200) uses the video stream monitored by the parking space monitoring module (100) as an input layer and the boundary box, confidence and category probability of the vehicle as an output layer to build a YOLO convolutional neural network model, uses a professional image annotation tool LabelImg to annotate the vehicles in the image extracted frame by frame from the video stream, uses a mean square error loss function to measure the difference between the predicted result and the actual annotation, trains the YOLO convolutional neural network model, inputs the image preprocessed by the parking space monitoring module (100) into the YOLO convolutional neural network model, determines whether the vehicle is parked in the parking space, and then extracts the license plate, front, rear and wheel features of the vehicle, and transmits the feature data of the vehicle to the data processing module (300); The data processing module (300) receives the vehicle characteristic data transmitted by the model building module (200), stores the data, records the vehicle entry time, and starts calculating the parking fee according to the preset charging rules. When the vehicle is about to leave the parking space, the parking space monitoring module (100) collects the vehicle image again, extracts the vehicle characteristic data through the model building module (200), and compares the data with the vehicle characteristic data stored when parking to determine whether the vehicle is the same vehicle. When the four wheels of the vehicle are not in the parking space, the charging is stopped, and a corresponding bill is generated according to the charging rules, and the bill is transmitted to the ETC payment module (400); The ETC payment module (400) detects that the parked vehicle has left the parking space, and automatically triggers the ETC antenna to start radio frequency scanning of a target area within a certain range near the parking space without delay in the same device system, detects a credible OBU, and compares it with the current vehicle license plate. After the comparison is consistent, a deduction is initiated. After the deduction is successful, the national or provincial ETC center sends a text message to notify the car owner of the parking consumption this time.

2. The integrated machine for automatic identification and evidence collection of on-street parking and ETC payment according to claim 1 is characterized by: The model building module (200) includes a training model unit (210) and a feature extraction unit (220); The training model unit (210) labels vehicles in an image data set collected from vehicles parked in parking spaces, divides the data set into a training set, a validation set and a test set according to a ratio, trains the model with a minimum mean square error loss function as a target, and adaptively adjusts the learning rate through an Adam optimizer, so that the training process is more stable and efficient; The feature extraction unit (220) uses a camera with an 8-megapixel high-definition video stream to take photos of the license plate, front, rear and wheels of the vehicle parked in the parking space, and extracts the features of the license plate, front, rear and wheels of the vehicle.

3. The integrated machine for automatic identification and evidence collection of on-street parking and ETC payment according to claim 2 is characterized by: When training the model, the training model unit (210) performs data enhancement operations on the image, flips the image horizontally and vertically, adds Gaussian noise, and then performs training again.

4. The integrated machine for automatic identification and evidence collection of on-street parking and ETC payment according to claim 2 is characterized by: The training model unit (210) removes redundant bounding boxes through a non-maximum suppression algorithm, selects the bounding box that is closest to the real car bounding box and has the highest confidence, and removes other bounding boxes that overlap more and are relatively inaccurate, thereby obtaining a clear and accurate car target detection result.

5. The integrated machine for automatic identification and evidence collection of on-street parking and ETC payment according to claim 2 is characterized by: The feature extraction unit (220) comprises a license plate recognition component (221) and a shape recognition component (222); The license plate recognition component (221) locates the position of the license plate through color space conversion and threshold segmentation, and then uses an edge detection algorithm to obtain edge information of the license plate to accurately locate the license plate; The shape recognition component (222) extracts features of the front, rear and wheels of the vehicle through a texture recognition method and a Hough circle transformation method.

6. The integrated machine for automatic identification and evidence collection of on-street parking and ETC payment according to claim 5 is characterized by: After locating the license plate, the license plate recognition component (221) uses a character segmentation and recognition algorithm to recognize the license plate number.

7. The integrated machine for automatic identification and evidence collection of on-street parking and ETC payment according to claim 1 is characterized by: The parking space detection module (100) comprises a parking determination unit (110) and a preprocessing unit (120); The parking determination unit (110) compares the vehicle area in the differential image with the parking space area based on the obtained differential image, and if the extracted vehicle area position overlaps with the parking space area, it can be preliminarily determined that the vehicle begins to enter the parking space; When preprocessing an image, the preprocessing unit (120) uses a median filtering method to replace the central pixel with the median of the neighboring pixels to remove salt and pepper noise.

8. The on-street parking automatic identification and evidence collection and ETC payment integrated machine according to claim 1, characterized in that: The parking space monitoring module (100) is provided with a parking time determination mechanism. When the four wheels of the vehicle are all inside the parking space, a certain number of frames are continuously observed. If the wheel positions of the vehicle do not change significantly during this period, it is finally determined that the vehicle has entered the parking space.

9. The integrated machine for automatic identification and evidence collection of on-street parking and ETC payment according to claim 1, characterized in that: When the data processing module (300) compares the extracted vehicle feature data with the vehicle feature data stored when the vehicle is parked, it uses a weighted fusion method to perform weighted fusion on the license plate features and the vehicle appearance features to determine whether the vehicles are the same.

10. The integrated machine for automatic identification and evidence collection of on-street parking and ETC payment according to claim 1, characterized in that: The ETC payment module (400) adopts multi-threaded communication, utilizes an event-driven architecture to process various events in the system, and uses a message queue to transmit events and data.

Images