A vehicle parking recognition method based on edge intelligence and roadside high-position monitoring video

Abstract

The present invention discloses a vehicle parking recognition method based on edge intelligence and roadside high-position monitoring video. First, a real-time video of the parking area is collected by a camera as the input for detection, and the processing of the video stream is completed in the edge terminal. Secondly, the position of the vehicle and the license plate in the video is extracted by a deep learning network, the license plate number is extracted by character recognition technology, and the vehicle and the corresponding license plate are bound in real time by combining network models. Then, dynamic tracking is performed according to the position coordinates between the upper and lower frame images, and the relative position of the vehicle and the parking space is determined by the correspondence of three-dimensional and two-dimensional coordinates. Finally, a corresponding judgment is made on whether the vehicle is parked based on the above recognition results. The present invention improves the recognition rate of vehicle parking behavior in roadside parking scenarios and the robustness of the algorithm, reduces the amount of calculation, and can accurately determine the parking behavior of the vehicle based on the vehicle and license plate position information. It can be used in the field of intelligent transportation and has a large promotion prospect.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation systems, and in particular to a vehicle parking recognition method based on edge intelligence and roadside high-position monitoring videos. Background Art

[0002] In recent years, the number of motor vehicles in my country has continued to grow. According to statistics from the Traffic Management Bureau of the Ministry of Public Security, as of June 2021, my country has 384 million motor vehicles, of which 292 million are cars, accounting for 76.04% of the total, and their number is still growing. Parking has begun to become a difficult problem that some cities continue to solve. The planning of roadside parking spaces can alleviate the pressure of parking to a large extent and has been promoted in many cities. However, at this stage, the supervision of roadside parking spaces in many cities in my country is still mainly based on manual management. This management method is very inefficient and consumes a lot of manpower costs. At the same time, it is difficult to manage during peak parking periods, and it is easy to charge wrong fees and random fees. Therefore, there is an urgent need for an efficient way to manage roadside parking spaces.

[0003] At present, the management schemes for roadside parking spaces mainly include parking meter-based, geomagnetic-based and video-based parking space management schemes. Both of these schemes require the installation of monitoring equipment on the roadside or on the road surface, which has high deployment costs and will damage the roadside. They are difficult to promote in some areas with well-equipped facilities. The vision-based management scheme can circumvent these problems and has low deployment costs, but some problems such as occlusion and large tilted shooting angles in roadside parking scenarios lead to low parking recognition accuracy, and the network bandwidth cost required for video stream processing in the cloud is high. Based on this problem, a vehicle parking recognition method based on edge intelligence and roadside high-position monitoring video is proposed. Edge intelligence technology is used for video stream processing, and various influencing factors in actual roadside parking scenarios are considered, which improves the recognition rate of effective vehicle parking orders and has high robustness. Summary of the invention

[0004] The present invention combines edge intelligence technology with machine vision technology, performs video stream processing in the edge terminal, and solves the problem of low vehicle parking recognition rate caused by factors such as occlusion, large angle distortion, and variable lighting that often appear in roadside high-position surveillance videos, thereby improving the robustness for various scenarios.

[0005] The object of the present invention is achieved through the following technical solutions: A vehicle parking recognition method based on edge intelligence and roadside high-position monitoring video, comprising the following steps:

[0006] 1) The edge device obtains the video stream of the parking area collected by the high-position roadside surveillance camera; extracts the vehicle position recognition results and license plate position recognition results in the video through the target detection network for identifying the vehicle position and the target detection network for identifying the license plate position, and recognizes the license plate number through the character recognition model;

[0007] 2) For each frame of the video, the output result of the target detection network that identifies the vehicle position is used as the input of the target detection network that identifies the license plate position. The binding information of the vehicle and its corresponding license plate is obtained based on the output result of the target detection network that identifies the license plate position, and the identified information is recorded in the edge device.

[0008] 3) The vehicle position recognition result of each frame image is calculated to coincide with the vehicle position recognition result of the previous frame image. Based on the coincidence calculation result, if it is determined that the vehicles in the two frames are the same vehicle, the vehicle position information of the previous frame image is updated.

[0009] 4) The parking space area in the video is expanded into a cube by adding height information, and then the four outermost boundary points of the cube are connected into a rectangular frame, which is compared with the recognition frame obtained based on the vehicle position recognition result to determine the relative position relationship between the vehicle and the parking space.

[0010] 5) Based on the identification and judgment results of steps 2)-4), a vehicle behavior recognition algorithm is used to determine whether the vehicle has entered or left the parking space, and the judgment result is reported to the cloud.

[0011] Furthermore, in step 1), each roadside high-position surveillance camera monitors 4-8 parking spaces, and the acquired video stream is processed in the local edge device. The target detection technology in computer vision is used to extract the vehicle and license plate position information, and the character recognition model is used to recognize the license plate number.

[0012] Furthermore, in step 2), the neural network model for vehicle position recognition is combined with the neural network model for license plate position recognition, the output result of the former is used as the input of the latter, and the recognition results of the two are stored in a container, and the vehicle and its corresponding license plate information are bound at the same time.

[0013] Furthermore, in step 3), the update of the vehicle position information takes the overlap between the vehicle position identification frames of the upper and lower frame images as the criterion. If the overlap is greater than the set threshold, the corresponding vehicle position information will be updated. If it is less than the threshold, the traversal will continue until all vehicles in the container are traversed. If none of the vehicles in the container are successfully matched, it is considered that the vehicle is entering the monitoring area for the first time, and new vehicle information will be created in the container. For the unmatched vehicle information in each frame of the image, a counting operation is performed on it. When the number of consecutive unmatched frames exceeds the set value, the information of this vehicle will be deleted. If the vehicle has been determined to be parked in a parking space, the information of this vehicle will always be saved until the vehicle leaves the parking space.

[0014] Furthermore, the relative position relationship between the vehicle and the parking space is determined as follows: the two-dimensional information in the surveillance video is expanded into three-dimensional information for calculation, the quadrilateral area of ​​the parking space is expanded into a cube by adding the height information corresponding to the vehicle, and then the four outermost boundary points of the cube are connected into a rectangular frame, and the overlap between the rectangular frame and the identification frame obtained according to the vehicle position recognition result is calculated, and two high and low thresholds are set for judgment. If it is greater than the high threshold, it is considered that the vehicle has entered the parking space, and if it is lower than the low threshold, it is considered that the vehicle has not yet parked in the parking space or left the parking space. If the overlap is between the two, the previous judgment result is maintained and the current recognition is invalid.

[0015] Furthermore, the vehicle behavior recognition algorithm uses the result of the relative position relationship judgment between the vehicle and the parking space as a basis, and obtains the behavior of entering and leaving according to the change of the judgment result. Furthermore, when the result of the relative position relationship judgment between the vehicle and the parking space changes, the vehicle behavior recognition algorithm starts timing while traversing the container. After reaching the set time threshold, the parking occupancy of each parking space is judged again. If it is first judged that the vehicle may leave the parking space, it will judge whether the corresponding parking space is vacant during the second judgment. If the two judgment results are consistent, it is confirmed that the vehicle has left. If the two judgment results are inconsistent, it will be considered that this recognition is invalid and the original recognition result is maintained; if it is first judged that the vehicle may enter the parking space, it will judge whether the corresponding parking space is vacant during the second judgment. If the two judgment results are consistent, it is confirmed that the vehicle has entered. If the two judgment results are inconsistent, it will be considered that this recognition is invalid and the original recognition result is maintained.

[0016] The beneficial effects of the present invention are as follows: in the present invention, the edge device processes the video stream information and judges the vehicle behavior, and only the recognition result is subsequently uploaded. Reduce the bandwidth cost consumed by local video stream transmission. Using the output of the vehicle position recognition network as the input of the license plate recognition network, the search range of the license plate recognition network model is reduced, the amount of calculation is reduced, the accuracy is improved, and based on this operation, the full binding of vehicle and license plate information is achieved. Using the overlap of the vehicle recognition frame of the upper and lower frame images as a measurement standard, dynamic tracking of the vehicle position is achieved, ensuring that the vehicle information is not lost. By combining two-dimensional and three-dimensional information, the problem of difficulty in judging the relative position of the vehicle and the parking space caused by the influence of the shooting angle is solved. Considering the factors that may affect the behavior judgment in actual roadside parking, the reliability of the judgment of the vehicle entering and leaving the parking space is improved by triggering and timing reconfirmation. Overall, the effective order recognition rate of the visual-based roadside parking management system and its robustness under various influencing factors are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall structural block diagram of the present invention.

[0018] Figure 2 It is a flow chart of vehicle and license plate information extraction of the present invention.

[0019] Figure 3 It is a vehicle position information updating flow chart of the present invention.

[0020] Figure 4 It is a flow chart for determining the relative position relationship between a vehicle and a parking space of the present invention.

[0021] Figure 5 It is a flow chart of vehicle behavior recognition of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described below in conjunction with the accompanying drawings.

[0023] like Figure 1 As shown, the present invention extracts and manages the video stream near the roadside parking spaces through the roadside high-position monitoring camera. Each roadside high-position monitoring camera monitors 4-8 parking spaces. The video stream is processed by the embedded device at the edge (such as a single-chip microcomputer, FPGA, etc.), including the extraction of vehicle and license plate position and license plate number information through computer vision related technologies. Subsequently, according to the extracted information, four steps are performed, namely, vehicle and license plate information binding, vehicle position information update, vehicle parking space relative position relationship judgment, and vehicle behavior recognition. Finally, information such as whether the vehicle is parked and the license plate number of the vehicle parked in the parking space, action time, corresponding parking space number, etc. is obtained. This information can be uploaded to the cloud service platform for subsequent processing.

[0024] like Figure 2 As shown in the figure, the process of extracting vehicle and license plate information through computer vision technology is as follows: first, the target detection network (such as YOLO, SSD, etc.) that identifies the vehicle position is used to obtain the identification frame of the vehicle position. Then, according to the specific coordinates of the identification frame, the area of ​​the vehicle in the image is segmented. After that, the segmented image is used as the input of the license plate positioning model to obtain the specific position of the license plate, and the binding information of the vehicle and its corresponding license plate is obtained, and the identified information is recorded in the edge device. Then, the license plate number is recognized based on the license plate area image obtained by image segmentation through character recognition models such as LPRNet. Finally, repeat the above operations, traverse all vehicle identification areas, obtain the information of the vehicle and license plate position and license plate number, and perform subsequent parking behavior recognition based on this.

[0025] like Figure 3 As shown in the figure, the vehicle position information update method is as follows: for each vehicle identification frame in each frame image, the vehicle position coordinates in the container will be traversed, and the overlap between the two will be calculated. The overlap is measured by the intersection-and-union ratio method, that is, the ratio of the intersection and union of the vehicle identification frames in the two frames is calculated. The overlap threshold is set according to the number of frames of the video stream and the actual processing speed. If the overlap is greater than the threshold, the corresponding vehicle position information will be updated. If it is less than the threshold, the traversal will continue until all vehicles in the container are traversed. If none of the vehicles in the container are successfully matched, it is considered that the vehicle enters the monitoring area for the first time, and new vehicle information is created in the container. For the unmatched vehicle information in each frame image, it is counted. When the number of consecutive unmatched frames exceeds 10 frames, the information of this vehicle will be deleted. However, if the vehicle has been determined to be parked in a parking space, the information of this vehicle will always be saved until the vehicle leaves the parking space.

[0026] like Figure 4 As shown in the figure, the process of determining the relative position relationship between the vehicle and the parking space is as follows: expand the two-dimensional information in the surveillance video into three-dimensional information for calculation, set the height information according to the camera shooting height, angle and specific needs, add the quadrilateral area of ​​the parking space to this height information to expand it into a cube, and then take the four outermost boundary points of the cube to connect into a rectangular frame to obtain the parking space area. The vehicle identification frame calculates the overlap between the parking space area one by one, and takes the maximum value for threshold judgment. Set two thresholds MAX and MIN as judgment criteria. When the overlap is greater than MAX, it is considered that the vehicle has parked in the parking space and output the number of the corresponding parking space. When the overlap is between MAX and MIN, it is in a suspicious state. At this time, the vehicle judgment result remains unchanged and -1 is output to prevent the vehicle identification frame position from shaking due to the recognition algorithm and environmental factors, and avoid erroneous results of vehicle recognition. When the overlap is less than MIN, the device will determine that the vehicle has not parked in any parking space or left the parking space, and output 0.

[0027] like Figure 5 As shown in the figure, the method of vehicle behavior recognition is as follows: first, the parking judgment result of each vehicle is recorded, and the output results of the vehicle parking judgment algorithm of the current frame and the previous frame image are compared. If the output value changes, for example, the output value changes from P0 to P1, P0 and P1 are the output results of the relative position relationship between the vehicle and the parking space. At this time, P0 and P1 may be 0 or a corresponding parking space number. If P0 is not 0, it is judged that the vehicle may have the behavior of driving out of the parking space corresponding to P0. Similarly, if P1 is not 0, the vehicle may have entered the parking space corresponding to P1. After that, it will be judged whether P0 is idle and whether P1 is occupied, so as to determine whether there is an action of the vehicle entering or leaving the parking space. The vehicle behavior recognition algorithm will record the time when each action occurs and report the data as the basis for generating parking orders in the subsequent cloud service platform. In order to improve the robustness of the vehicle behavior recognition algorithm and prevent behavior recognition errors caused by irregular parking of vehicles or parking judgment errors in a certain frame, when the output value of the vehicle parking judgment changes, the timing will start while traversing the container. After reaching the set time threshold of 60 seconds, the parking occupancy of each parking space will be judged again. If it is first judged that the vehicle may leave the parking space, it will be judged whether the corresponding parking space is vacant in the second judgment. If the two judgment results are consistent, it is confirmed that the vehicle has left. If the two judgment results are inconsistent, it will be considered that this recognition is invalid and the original recognition result is maintained; if it is first judged that the vehicle may enter the parking space, it will be judged whether the vehicle has parked in the parking space in the second judgment. If the two judgment results are consistent, it is confirmed that the vehicle has entered. If the two judgment results are inconsistent, it will be considered that this recognition is invalid and the original recognition result is maintained; thereby verifying the correctness of the behavior recognition algorithm results.

[0028] The present invention completes the processing of the video stream in the edge terminal, and does not need to transmit the video stream, thus saving network bandwidth. In addition, various factors affecting recognition in roadside parking scenarios are considered, and the vehicle and license plate information are extracted using computer vision technology to identify the vehicle parking behavior. This method has high recognition accuracy and strong robustness, and can be applied in various roadside environments.

[0029] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A vehicle parking recognition method based on edge intelligence and roadside high-position monitoring video, characterized in that: The following steps are involved: 1) The edge device obtains the video stream of the area near the parking space collected by the high-position roadside surveillance camera; The vehicle position recognition results and the license plate position recognition results in the video are extracted through the target detection network for identifying the vehicle position and the target detection network for identifying the license plate position, and the license plate number is recognized through the character recognition model; 2) For each frame of the video, the output result of the target detection network for identifying the vehicle position is used as the input of the target detection network for identifying the license plate position. The binding information of the vehicle and its corresponding license plate is obtained based on the output result of the target detection network for identifying the license plate position, and the identified information is recorded in the edge device; 3) The vehicle position recognition result of each frame image is calculated to coincide with the vehicle position recognition result of the previous frame image. According to the coincidence calculation result, if it is determined that the vehicles in the two frames are the same vehicle, the vehicle position information of the previous frame image is updated; 4) The parking space area in the video is expanded into a cube by adding height information, and then the four outermost boundary points of the cube are connected into a rectangular frame. The rectangular frame is compared with the recognition frame obtained based on the vehicle position recognition result to determine the relative position relationship between the vehicle and the parking space; 5) Based on the recognition and judgment results of steps 2)-4), the vehicle behavior recognition algorithm is used to judge whether the vehicle has entered or left the parking space, and the judgment result is reported to the cloud.

2. According to claim 1, a vehicle parking recognition method based on edge intelligence and roadside high-position monitoring video is characterized in that: In step 1), each high-position roadside surveillance camera monitors 4-8 parking spaces. The acquired video stream is processed in the local edge device, and the object detection technology in computer vision is used to extract the vehicle and license plate location information, and the character recognition model is used to recognize the license plate number.

3. According to claim 1, a vehicle parking recognition method based on edge intelligence and roadside high-position monitoring video is characterized in that: In step 2), the neural network model for vehicle position recognition is combined with the neural network model for license plate position recognition. The output of the former is used as the input of the latter, and the recognition results of the two are stored in a container. At the same time, the vehicle and its corresponding license plate information are bound.

4. The vehicle parking recognition method based on edge intelligence and roadside high-position monitoring video according to claim 3 is characterized in that: In step 3), the update of the vehicle position information takes the overlap between the vehicle position identification frames of the upper and lower frame images as the judgment standard. If the overlap is greater than the set threshold, the corresponding vehicle position information will be updated. If it is less than the threshold, the traversal will continue until all vehicles in the container are traversed. If no vehicles in the container are successfully matched, it is considered that the vehicle enters the monitoring area for the first time, and new vehicle information will be created in the container. For the unmatched vehicle information in each frame of the image, it is counted. When the number of consecutive unmatched frames exceeds the set value, the information of this vehicle will be deleted. If the vehicle has been determined to be parked in a parking space, the information of this vehicle will always be saved until the vehicle leaves the parking space.

5. The vehicle parking recognition method based on edge intelligence and roadside high-position monitoring video according to claim 1 is characterized in that: The relative position relationship between the vehicle and the parking space is determined as follows: the two-dimensional information in the surveillance video is expanded into three-dimensional information for calculation, the quadrilateral area of ​​the parking space is expanded into a cube by adding the height information corresponding to the vehicle, and then the four outermost boundary points of the cube are connected into a rectangular frame, and the overlap between the rectangular frame and the identification frame obtained according to the vehicle position recognition result is calculated. Two high and low thresholds are set for judgment. If it is greater than the high threshold, it is considered that the vehicle has entered the parking space, and if it is lower than the low threshold, it is considered that the vehicle has not yet parked in the parking space or left the parking space. If the overlap is between the two, the previous judgment result is maintained and the current recognition is invalid.

6. The vehicle parking recognition method based on edge intelligence and roadside high-position monitoring video according to claim 3 is characterized in that: The vehicle behavior recognition algorithm uses the result of the relative position relationship judgment between the vehicle and the parking space as a basis, and obtains the entry and exit behavior according to the change of the judgment result; further, when the result of the relative position relationship judgment between the vehicle and the parking space changes, the vehicle behavior recognition algorithm starts timing while traversing the container. After reaching the set time threshold, the parking occupancy situation of each parking space is judged again. If it is first judged that the vehicle may leave the parking space, it will judge whether the corresponding parking space is vacant during the second judgment. If the two judgment results are consistent, it is confirmed that the vehicle has left. If the two judgment results are inconsistent, it will be considered that this recognition is invalid and the original recognition result is maintained; if it is first judged that the vehicle may enter the parking space, it will judge whether the corresponding parking space is vacant during the second judgment. If the two judgment results are consistent, it is confirmed that the vehicle has entered; if the two judgment results are inconsistent, it will be considered that this recognition is invalid and the original recognition result is maintained.

Images