Vehicle occupancy determination and license plate positioning method based on tof reflectivity and images

Abstract

The application discloses a vehicle occupancy judgment and license plate positioning method based on TOF reflectivity and images, and belongs to the technical field of license plate positioning. The method utilizes multi-point TOF data time sequence analysis and fuses RGB image vehicle occupancy judgment and license plate region of interest positioning. First, multi-point TOF data is utilized for distance measurement judgment. When the distance measurement result is stable and a threshold value of occupancy is triggered, TOF depth data, reflectivity data and RGB image data are utilized to form five-channel data, and a deep convolutional neural network is constructed based on a multi-task learning architecture. One branch of the task is vehicle detection, and another branch of the task is license plate region of interest positioning. Through time sequence analysis of TOF data, the judgment stability of scene change states is greatly enhanced. Subsequent high-power consumption calculation is triggered only when the scene changes and is stable, so that low-power consumption and high-frequency judgment of vehicle occupancy judgment are ensured.

Description

Technical Field

[0001] This invention relates to a method for vehicle occupancy determination and license plate location based on TOF reflectivity and images, belonging to the field of license plate location technology. Background Technology

[0002] Vehicle occupancy detection and license plate recognition are core supporting technologies for the Internet of Things (IoT) and smart cities. Currently, cameras are the primary technological means. Generally, deep learning methods are used to detect vehicles to determine occupancy, and then deep learning methods are used to detect the license plate area within the vehicle area to complete license plate localization. The localized license plate area is then handed over to a character recognition module to complete license plate recognition.

[0003] Currently, the main technical methods for vehicle occupancy detection can be divided into Time-of-Flight (TOF) based methods and camera-based methods.

[0004] Time-of-flight (TOF) based methods primarily use a single-point laser to measure distances based on the time of flight, and then determine whether the target location is occupied based on the ranging results. This method is low-cost and simple to implement. However, it is highly prone to false detections, especially in rainy weather. Furthermore, due to the lack of semantic information, occlusion by other objects, such as pets, trash, bicycles, tricycles, and electric vehicles, can also cause false detections.

[0005] Camera-based methods primarily use convolutional neural networks to detect vehicles in images and pre-label parking space areas within the image. The degree of overlap between the bounding box of the detected vehicle and the target area determines whether the parking space is occupied. While this method offers high accuracy, the frequent use of deep neural network inference significantly increases device power consumption, thus severely limiting its practical application.

[0006] In the localization of license plate regions in images, the main current method is to use deep neural networks for target detection. However, this method involves searching a large image area, while the license plate region often occupies a relatively small area. Therefore, it is computationally intensive and prone to missed detections.

[0007] How to perform vehicle occupancy judgment and license plate interest area localization at high frequency under low power consumption is currently a bottleneck problem and one of the main technical obstacles restricting the deployment of license plate recognition technology in parking space scenarios. Summary of the Invention

[0008] The purpose of this invention is to provide a method for vehicle occupancy determination and license plate location based on TOF reflectivity and images, so as to solve the problems raised in the prior art.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] This paper proposes a method for vehicle occupancy detection and license plate localization based on Time-of-Flight (TOF) reflectance and image analysis. It utilizes multi-point TOF data for time-series analysis and integrates RGB images for vehicle occupancy detection and license plate region of interest (ROI) localization. First, it uses multi-point TOF data for ranging detection. When the ranging result is stable and the occupancy threshold is triggered, it uses TOF depth data, reflectance data, and RGB image data to form a five-channel dataset. Based on a multi-task learning architecture, a deep convolutional neural network is constructed, with one branch for vehicle detection and another branch for license plate ROI localization.

[0011] As a preferred example, the specific steps include the following:

[0012] 1) First, based on multi-point TOF data, calculate the change confidence of the interest space according to the 3D placeholder method. The calculation method is the proportion of 3D placeholders that have changed to the total number of placeholders.

[0013] 2) Set a threshold for the probability of scene change, and make subsequent judgments for scenes with a probability of change greater than 0.5. Calibrate the TOF ranging data and RGB image data, and project the ranging data and reflectance data onto the imaging plane of the RGB camera to form data of five channels of RGBDI, where R-red, G-green, B-blue, D-range, and I-reflectance.

[0014] 3) Design a multi-task learning convolutional neural network. The network output includes two parts: the 2D bounding box of the vehicle and the 2D bounding box of the license plate area. Make full use of the difference in target background reflectivity caused by the color and paint of the vehicle body and license plate. The loss function is the intersection-union ratio of the vehicle bounding box and the license plate bounding box with the ground truth. The backbone network of the neural network uses MobileNetV2. Due to the introduction of D and I, fewer network layers and fewer channels per layer are selected. The width factor and resolution factor are both set to 0.25, and the number of network layers is set to 1 / 2 of the standard MobileNetV2.

[0015] 4) Use a multi-task network to perform integrated judgment of vehicle position and license plate location. If no vehicle is detected, the license plate location is not output; if a vehicle is detected, the license plate location is output.

[0016] As a preferred example, in step 1), the criterion for judging the change of the 3D placeholder is: the ratio of the placeholder reflectance data at two time points is greater than 1.5.

[0017] The beneficial effects of this invention are:

[0018] 1. By analyzing the time series of TOF data, the stability of the judgment of scene change status is greatly enhanced. Subsequent high-power calculations are only triggered after the scene changes and stabilizes, ensuring low-power, high-frequency judgment of vehicle occupancy.

[0019] 2. Time series analysis of multi-point TOF data, compared with single-point ranging data, can greatly improve the ability to resist interference, effectively suppress random interference from rain and strong light, improve the stability of scene change judgment, and eliminate misjudgment of scene changes caused by local random changes.

[0020] 3. The addition of a multi-task learning architecture and TOF reflectivity can greatly improve the robustness of deep neural networks for vehicle detection and license plate localization. Furthermore, the introduction of multi-source information can significantly reduce the width and depth of the network, thereby greatly reducing computational load and power consumption. Attached Figure Description

[0021] Figure 1 This is a flowchart of the calculation scheme of the present invention. Implementation

[0022] To facilitate a clear understanding of the technical means, inventive features, objectives, and effects of this invention, the following description is provided in conjunction with... Figure 1 The specific process of the above method will be explained as follows:

[0023] In step 401, the RGB image and multi-point TOF data are first geometrically calibrated and fused. Using a calibration board with obvious depth undulations and color changes and known geometric information, the image and multi-point TOF information are obtained. The two are aligned, and the relative position and pose difference between the camera and the TOF data are calculated to obtain the transformation matrix. The ranging information and reflectivity information of the TOF data are projected onto the RGB plane using the transformation matrix to form 5-dimensional fused information with RGB colors + depth + reflectivity. Then, proceed to step 402.

[0024] In step 402, a multi-task neural network is trained using MobileNetV2 as the backbone and the Intersection over Union (IoU) of the 2D bounding boxes of vehicles and license plates with the ground truth values ​​as the loss function. The network's width and resolution factors are both set to 0.25. The network depth is half that of standard MobileNetV2. The trained neural network is then used in other steps.

[0025] Step 403: Divide the target space into 5cm × 5cm space occupancy grids. Calculate the occupancy status of each space occupancy grid every 1 second. The ratio of the number of changing grids to the total number of grids is used as the probability of change. The calculation method for whether each grid has changed is as follows: calculate the sum of the reflectance of that grid. If the ratio of the sum of reflectance at two different times is less than 1.5, it is considered that the grid has not changed its occupancy; otherwise, it is considered that it has changed its occupancy. If the probability of change is less than 0.5, repeat step 403; if the probability of change is greater than 0.5, proceed to step 404.

[0026] Using the image calibrated in step 401 and the multi-task neural network trained in step 402, vehicles and license plate regions are detected simultaneously. If a vehicle is detected and the license plate region is within the vehicle region, the result is output; otherwise, the detection result is ignored.

[0027] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention, all of which fall within the scope of the claims. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A vehicle occupancy determination and license plate localization method based on TOF reflectivity and images, characterized in that: Vehicle occupancy judgment and license plate region positioning are performed by using multi-point TOF data timing analysis and fusing RGB images. First, multi-point TOF data is used for distance measurement judgment. When the distance measurement result is stable and the occupancy threshold is triggered, TOF depth data, reflectivity data and RGB image data are used to form five channels of data. Based on a multi-task learning architecture, a deep convolutional neural network is constructed. One branch of the task is vehicle detection, and the other branch of the task is license plate region positioning. Specifically, the method comprises the following steps: 1) First, based on multi-point TOF data, the change confidence of the interested space is calculated in the 3D occupancy grid mode. The calculation method is that the proportion of the changed 3D occupancy grid in the total occupancy grid is calculated; 2) A threshold is set for the scene change probability. For the scene with a change probability greater than 0.5, subsequent judgment is performed. TOF distance measurement data and RGB image data are calibrated. The distance measurement data and reflectivity data are projected onto the imaging plane of the RGB camera to form five channels of RGBDI data, wherein R-red, G-green, B-blue, D-distance, and I-reflectivity; 3) A multi-task learning convolutional neural network is designed. The output of the network includes two parts: the 2D bounding box of the vehicle and the 2D bounding box of the license plate region. The color and paint of the vehicle body and license plate are fully utilized to cause the target background reflectivity difference. The loss function is the intersection over union of the vehicle bounding box and the license plate bounding box and the true value. The backbone network of the neural network uses MobileNetV2. Due to the introduction of D and I, fewer network layers and channel numbers of each network layer are selected. The width factor and the resolution factor are both set to 0.25, and the network layer number is set to 1 / 2 of the standard MobileNetV2. 4) The multi-task network is used for integrated judgment of occupancy and license plate position. If no vehicle is detected, the license plate position is not output. If a vehicle is detected, the license plate position is output.

2. The TOF reflectivity and image based vehicle occupancy judgment and license plate localization method of claim 1, wherein: In step 1), the standard for judging the change of the 3D occupancy grid is that the ratio of the reflectivity data of the occupancy grid at two time points is greater than 1.5.

Images