A parking lot parking space occupation state panoramic detection method and system

Abstract

The application discloses a kind of parking lot berth occupation state panoramic detection method and system, comprising: obtaining the spatial layout form of the parking lot to be managed, according to the coverage range of each monitoring camera, determine the layout scheme of the camera required to realize parking lot berth detection of the parking lot to be managed;Construct berth state detection module, for obtaining the vehicle and parking space information in parking lot;Construct panoramic base map generation module, specifically end-to-end perspective transformation model, for generating the 2D panoramic base map of the shooting object;Construct original digitized 2D parking lot panoramic map;According to the real-time obtained vehicle and vehicle information in parking lot, generate the visual representation of berth real-time detection result, namely real-time digitized 2D parking lot panoramic berth occupation base map.The application requires less equipment, can be flexibly deployed and applied in the management and operation work of different forms of outdoor parking lot, and provides panoramic berth occupation visual analysis, reduces management cost and improves management efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, in particular to a parking lot occupancy state panoramic detection method and system. BACKGROUND

[0002] At present, in the application of parking lot sensing technology, the existing video monitoring detection management method gradually replaces the sensing and detection means based on radar, infrared and microwave, etc. due to its more robust anti-interference ability, better recognition accuracy and stronger visual expression ability, and is widely and universally applied in parking lot management. However, whether it is a parking lot manager or a driving user, the "information island" parking problem is still mostly caused by the extensive, non-real-time and unreliable sensing of real-time available parking resources in the parking lot, which brings great obstacles to the further utilization of limited and tight parking resources. In order to solve the above problems, a video monitoring camera can be installed at each parking space to sense the parking occupancy state, but the management cost of this method is undoubtedly huge, which is difficult to further popularize and apply.

[0003] Therefore, it is necessary to design a new video monitoring parking lot management method and system to intelligently and less expensively dredge and speed up the flow of information and data of this basic link of urban parking. SUMMARY

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of the present application is to provide a parking lot occupancy state panoramic detection method and system.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A parking lot occupancy state panoramic detection method, comprising:

[0007] Obtaining the spatial layout form of the parking lot to be managed, determining the layout scheme of the camera required to realize the parking lot detection of the parking lot to be managed according to the coverage range of each monitoring camera;

[0008] Constructing a parking state detection module, the parking state detection module comprising a vehicle recognition model based on deep learning and a parking space recognition model based on deep learning, for obtaining vehicle and parking space information in the parking lot;

[0009] Constructing a panoramic base map generation module, specifically an end-to-end perspective transformation model, the end-to-end perspective transformation model comprising a perspective transformation model and a visual stitching model, for generating a 2D panoramic base map of the photographed object;

[0010] The original digital 2D parking lot panoramic image is constructed, the initialization parking space information under each monitoring perspective image is obtained through a vehicle recognition model and a parking space recognition model, an end-to-end perspective transformation model is run to splice and fuse the image sequence including the initialization parking space information under each perspective to form the original digital 2D parking lot panoramic image;

[0011] According to the real-time obtained vehicle and vehicle information in the parking lot, the real-time parking space occupation state under each monitoring perspective is obtained through a perspective transformation model and a visual splicing model, the original digital 2D parking lot panoramic image is fused to generate the visual representation of the real-time detection result of the parking space, that is, the real-time digital 2D parking lot panoramic parking space occupation base map.

[0012] Further, the vehicle recognition model adopts a lightweight improved YOLOv4 network as a detection framework, specifically:

[0013] Three scale feature maps of the input image are extracted by using a backbone network, and the backbone network is a CSP DarkNet53;

[0014] The three scale feature maps extracted by the feature fusion network are sequentially fused according to the scale size and the fusion weight decreases;

[0015] The feature maps output by the feature fusion network are subjected to convolution operation to obtain three feature vectors of different lengths, and the target information in the input image is obtained by decoding the feature vectors, and the target information includes target type and position.

[0016] Further, the parking space recognition model adopts a lightweight SSD network as a detection framework, specifically:

[0017] The input image is subjected to feature extraction by using a backbone network, and the backbone network is a ResNet-50;

[0018] The target information in the input image is obtained by fusing the features by using a pruned SSD network, and the pruned SSD network only retains the network structure of the output 10*10 size feature map.

[0019] Further, the perspective transformation model adopts VGG-16 as a feature extraction backbone network in the training stage, the perspective transformation model takes a pair of images satisfying a certain perspective transformation relationship as input, outputs the offset of four feature points between the two images, fuses the offset with the original four feature point coordinate information, and can solve the coordinates of the corresponding feature points before and after transformation, thereby obtaining the perspective transformation between the image pair.

[0020] Further, the construction of the original digital 2D parking lot panoramic image has the following working process:

[0021] The vehicle recognition model and the vehicle recognition model are called in sequence to detect the vehicle parking positions and the empty parking spaces under each monitoring view angle;

[0022] The position of the vehicle parking and the position of the parking space have a one-to-one correspondence relationship, a dynamic background generation method is adopted, pixel coordinates of the vehicle parking position and the empty parking space position are iteratively traversed in a sliding window manner, and a moving average method is used to calculate weighted mean values of the vehicle parking position and the empty parking space position information in each iteration until the weighted mean values converge.

[0023] The converged weighted mean values can represent the parking space positions in a single monitoring video view angle, the pixel coordinates of the center points of each parking space in the image are determined by the values, and the boundaries of each parking space are drawn by a quadrilateral frame with a set color, thereby generating unique position identification information of each parking space, which includes pixel coordinates of four vertices of the boundary frame corresponding to each parking space position in the image, pixel coordinates of the center point of the boundary frame, and a unique ID number, which are in a one-to-one correspondence relationship. The above information is superimposed and drawn to the original monitoring video image to obtain a composite image, which is a single monitoring camera imaging image with the parking space position information drawn, and the single monitoring camera imaging image with the parking space position information drawn is input into an end-to-end view conversion model to generate an original digital 2D parking lot panoramic base map, which presents the splicing result of a plurality of "single monitoring camera imaging images with the parking space position information drawn" in a vertical bird's eye view angle, and the splicing result contains ID numbers, boundary frame vertex coordinates and boundary frame center point coordinates of all parking spaces in the field.

[0024] Further, according to the real-time obtained vehicle and vehicle information in the parking lot, the real-time parking space occupation state under each monitoring view angle is obtained through a perspective transformation model and a visual splicing model, and a real-time detection result of the parking space is fused to generate a visual representation of the parking space, i.e. a real-time digital 2D parking lot panoramic parking space occupation base map, which is specifically:

[0025] First, a nearest neighbor algorithm is used to search for all undetermined parking space ID numbers in the digital 2D parking lot panoramic image, whose Euclidean distance in the pixel coordinate system from the center point of the boundary frame of the real-time vehicle position detected by the vehicle detection model is less than a certain threshold a. These undetermined parking spaces are parking spaces that may have been used;

[0026] Further, the IOU judgment method is specifically:

[0034] The IOU judgment method is specifically: the relationship between the vehicle position bounding box obtained by the vehicle recognition model and the position information of the parking space contained in the single monitoring camera imaging image with the drawn parking space position information is determined one by one, the real-time occupancy of the parking space in all single camera monitoring visual angle images in the parking lot is determined, then the judgment result is fused with the information of the original digital 2D parking lot panoramic base map, and a visual representation of the real-time detection result of the parking space, that is, a real-time digital 2D parking lot panoramic parking space occupancy base map, is generated.

[0027] Further, the visual representation of the real-time detection result of the parking space, that is, the real-time digital 2D parking lot panoramic parking space occupancy base map, is specifically:

[0028] For each parking space ID, a 0-1 state indication identifier is set, and the identifier is identified as "1" for an occupied parking space and "0" for an empty available parking space. A list with a length of the maximum number of parking space IDs is created to store the state indication identifier values of all parking spaces. The state indication identifier values are associated with the information of the original digital 2D parking lot panoramic base map, and the occupied parking spaces and the empty available parking spaces are marked with different color markers on the original digital 2D parking lot panoramic base map. The result is the "real-time digital 2D parking lot panoramic parking space occupancy base map", which is a visual representation of the real-time occupancy of the parking space in the parking lot.

[0029] Further, the layout scheme of the camera required for the parking space detection of the parking lot to be managed is determined, and the layout scheme is specifically:

[0030] The minimum number of camera devices required and the position of each camera are determined with reference to the effective monitoring coverage range of the single camera and the spatial layout form of the parking lot.

[0031] Further, the visual stitching model is specifically:

[0032] First, the imaging images of the monitoring cameras are sequentially grouped into a sequence as input, and the corresponding new image sequence with a vertical bird's eye view as a visual angle is output. Then, the new image sequence is stitched in a panoramic manner by using the SIFI feature point matching stitching method to generate a 2D panoramic base map of the corresponding parking lot.

[0033] Further, the IOU judgment method is specifically:

[0034] The IOU judgment method is specifically: the relationship between the vehicle position bounding box obtained by the vehicle recognition model and the position information of the parking space contained in the single monitoring camera imaging image with the drawn parking space position information is determined one by one, the real-time occupancy of the parking space in all single camera monitoring visual angle images in the parking lot is determined, then the judgment result is fused with the information of the original digital 2D parking lot panoramic base map, and a visual representation of the real-time detection result of the parking space, that is, a real-time digital 2D parking lot panoramic parking space occupancy base map, is generated.

[0035] A parking lot occupancy state panoramic detection system, a plurality of monitoring cameras are arranged above a parking lot to be managed, the plurality of monitoring cameras cover the entire parking lot range, a parking space state detection module and a panoramic base map generation module are built in each monitoring camera, a vehicle recognition model and a parking space recognition model are automatically run on startup, initialization parking space information under each monitoring visual angle image can be automatically obtained, then a panoramic base map generation module is run to splice and fuse image sequences containing initialization parking space information under each visual angle, a panoramic 2D parking lot digital electronic base map is formed, and the panoramic 2D parking lot digital electronic base map is stored and shared to a management end data background.

[0036] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0037] 1、The present application adopts deep learning technology, which can accurately identify vehicles and parking spaces and infer the positions of vehicles and parking spaces under fixed visual angles;

[0038] 2、The present application adopts deep learning technology and image processing technology, which can accurately calculate the perspective transformation from images formed by a camera to new images with a vertical bird's-eye view as a visual angle, and realize panoramic splicing by using a feature point matching method;

[0039] 3、The present application adopts a dynamic background generation method, which can realize automatic digital 2D parking lot panoramic base map generation and reduce the cost of manual parking lot panoramic base map annotation;

[0040] 4、The present application adopts a two-stage algorithm combining nearest neighbor analysis and intersection over union (IOU) judgment method, which can accurately detect parking space occupancy state. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a workflow diagram of the present application;

[0042] Figure 2 is a schematic diagram of monitoring device layout and coverage range in an embodiment of the present application;

[0043] Figure 3 is a structural schematic diagram of a vehicle recognition model in an embodiment of the present application;

[0044] Figure 4 is a structural schematic diagram of a parking space recognition model in an embodiment of the present application;

[0045] Figure 5 is a structural schematic diagram of a perspective transformation model in an embodiment of the present application;

[0046] Figure 6 is a parking lot 2D panoramic electronic base map generation flowchart in an embodiment of the present application;

[0047] Figure 7This is a logical data flow diagram of each module in an embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.

[0049] In existing technologies, the "information silo" parking problem largely stems from the crude, non-real-time, and unreliable perception of real-time available parking spaces for both parking lot managers and drivers. This poses a significant obstacle to the further utilization of limited and scarce parking resources. To address this issue, video surveillance cameras could be installed in each parking space to monitor occupancy. However, this method is undoubtedly costly and difficult to promote further. Therefore, a new video surveillance parking space management method and system are needed to streamline and accelerate the flow of information and data in this most fundamental aspect of urban parking with greater intelligence and lower management costs.

[0050] The following is combined Figures 1-7 The present invention provides a detailed description of a panoramic detection system and method for parking space occupancy status based on deep learning and computer vision, using a specific embodiment.

[0051] Figure 1 The overall execution process of the algorithm is described. Specifically, it includes the following steps:

[0052] S110, Real-world environment survey: Investigate the spatial layout of the parking lot to be managed, and determine the camera layout scheme required to achieve full coverage of parking space detection in the parking lot to be managed based on the effective monitoring coverage of a single camera.

[0053] like Figure 2 As shown in the schematic diagram of the camera layout scheme in this embodiment, the present invention limits the maximum field of view covered by a single camera to approximately 20 parking spaces. Too large a coverage area will face serious boundary distortion and occlusion problems, negatively impacting the algorithm's performance. For parking lots of different shapes, the 2D spatial planes covered by the monitoring range of each camera must partially intersect. Furthermore, each camera's monitoring image should contain as many parking spaces as possible. This is fundamentally to ensure sufficient matching feature point pairs between images captured by adjacent cameras, avoiding excessive parallax in imaging from different viewpoints, thus ensuring successful image stitching and fusion to generate a more natural, aesthetically pleasing, and continuous panoramic map.

[0054] S120 constructs a parking space state detection module: screen the images with similar shooting angles and actual monitoring angles in the open source datasets UA-DETRAC dataset, VisDrone dataset and Parkinglot dataset as training data, and train two CNN models of vehicle recognition model and parking space recognition model based on deep learning under the pytorch framework. The two models of vehicle detection and parking space detection can independently complete the parking space detection task, or their output results can be supplemented as a comprehensive criterion for parking space occupation.

[0055] Figure 3 A specific embodiment of the vehicle recognition model based on YOLOv4 is given. The backbone network for extracting target feature information of the vehicle recognition model selects CSP DarkNet53, and the selected fusion network is the structure of SPP (Spatial pyramid pooling) + PAN (Path Aggregation Network), which fuses feature map information of three scales of 19x19, 38x38 and 76x76. Since larger scale feature maps provide more rich shallow feature information, which is helpful for the detection of small targets, in order to meet the application requirements under the characteristics of actual parking lot monitoring angle, it is necessary to improve the detection ability of small and medium scale vehicle targets, and more attention is paid to the proportion of information contained in large scale feature maps during judgment, so the fusion weights of three scale feature maps of 76x76, 38x38 and 19x19 decrease in turn. The trained model inputs real-time monitoring images with a resolution of 608x608, and detects vehicle targets in the images through the detection results of three different scale feature maps of 76x76, 38x38 and 19x19. The implementation idea of different weight fusion discrimination of different scale feature maps is to apply different threshold non-maximum suppression to the original detection results of three different scale feature maps. The smaller the scale of the detection box with the same confidence is, the more significant the suppression is, so the detection results of medium and large feature maps can be highlighted.

[0056] In the target classification detection and position regression process, the feature fusion network outputs three different scale size feature map information through a series of convolution operations, and three different length feature vectors are obtained. Decoding the feature vector can obtain the information of the target category, position and the like contained in the graph. In order to strengthen the detection rate of the vehicle target, on the one hand, the inter-class distance between different target categories is increased, and the center loss idea is introduced, the classification loss function is converted into center-crossentropy loss, and greater punishment is given to the inter-class distinction error; on the other hand, the appearance prior is introduced, and the detection result of the detection frame with a specific width-height ratio is focused on, and combined with the actual appearance size of the vehicle, the detection result of the detection frame with a certain ratio of 1:2, 2:3, 1:3 and its reciprocal output by the YOLOv4 algorithm can be investigated and tracked as appropriate, so as to reduce the false positive and false negative rate of detection.

[0057] That is, the feature fusion part of the existing YOLOv4 network adopts ordinary addition for feature fusion, and the improved YOLOv4 network considers the imaging size of the vehicle and the parking space under the high-position camera view, adopts weighted addition, the large feature map corresponds to the small target, and the small feature map corresponds to the large target, and the detection of the small target needs to be strengthened, and the detection of the large target needs to be reduced, so that the large feature map needs to be given a larger weight, and the small feature map needs to be given a smaller weight, which is "scale size fusion weight descending"

[0058] In the present application, the feature fusion network extracts three scale feature maps according to the scale size fusion weight descending.

[0059] As shown in Figure 4 , a schematic diagram of a parking space recognition model based on a lightweight SSD network. The parking space recognition model selects a lightweight SSD network as a detection framework, and a parking lot dataset is used as a training dataset, which covers parking space images taken under different weather conditions and different light conditions. The target feature information extraction backbone network is composed of ResNet-50, which is used for feature extraction of the input image. The features are relative to the downstream tasks (detection, classification, tracking), and these features are general. Different downstream tasks can fine-tune the features according to the specific task head. Traditional image features include feature points, edges and contours, etc.

[0060] In order to meet the application requirements under the actual parking lot monitoring view angle characteristics, since the information provided by the large-scale feature map helps to improve the detection ability of small and medium scale parking space targets, the original SSD network needs to be pruned, and only the network structure of the output 10*10 size feature map can be retained, and the subsequent network structure of the smaller feature map is not needed. In the implementation, the detection results of 38*38, 19*19 and 10*10 three different scale feature maps are selected for weighted fusion. The implementation idea is that different threshold non-maximum suppression is applied to the original detection results of the three different scale feature maps, and the detection frame with the same confidence is more significantly suppressed in the smaller scale feature map, so that the detection results of the medium and large feature maps can be highlighted.

[0061] S130 constructs a panoramic base map generation module, specifically an end-to-end view transformation model, which includes a perspective transformation model and a visual stitching model, for generating a 2D panoramic base map of the photographed object.

[0062] As shown in Figure 5 , the perspective transformation model in the embodiment is a very key link for generating a panoramic electronic base map of the parking lot. The model selects VGG-16 as a feature extraction backbone network, and the training data set is a pair of images that satisfy a certain perspective transformation relationship. These image pairs used for training can be artificially generated from a single source image by using a “local sampling picture boundary point small perturbation method”.

[0063] Further, the local sampling picture boundary point small perturbation method specifically includes:

[0064] The input of this model is an image pair P A , P B consisting of two 128*128 gray images, wherein P A is a local image cut from a source image I A ; P B is a local image cut from I B which has a perspective transformation relationship with P A . The specific method is: first, taking the pixel coordinate system as the reference, starting from a certain starting point, cutting out a fixed size (128*128) square original local image P A from the single source image I A ; then performing random perturbation, randomly perturbing the four vertices of the cut original local image to obtain a new local image in the form of an arbitrary quadrilateral, thereby forming a source image pair with a specific perspective transformation relationship with the original local image. Subsequently, the second cutting is performed, and the perspective transformation matrix between the image pair is calculated, and the inverse of the perspective transformation matrix is multiplied with the original input image to obtain I B, in the first cutting start point of cutting, again from I B , the size of the square local image P (128x128) is cut out B , the two square local images obtained by the first cutting and the second cutting are spliced to obtain an image pair for model training that satisfies the perspective transformation relationship. It is worth noting that I A , I B satisfies the relationship of I A = H 0 AB I B .

[0065] The intermediate output of the model is the offset of 4 feature points between the two images (an 8x1 vector), and the output offset is fused with the 4 vertex coordinate information to obtain 4 pairs of corresponding vertex coordinates before and after transformation. Using these coordinates, a perspective transformation linear equation set is constructed, and the estimate H AB of the perspective transformation between the image pair is solved by the DLT (Direct Linear Transform) module. A After parameterized sampling window sampling and differentiable sampling operations, the source image I A is obtained. A Then, the estimated perspective transformation is applied to I' A to obtain I" A , I" B and I AB are used to construct a pixel-by-pixel reconstruction error loss function, and the gradient of the loss function is backpropagated to make the estimated H 0 of the perspective transformation model approach H AB . A , I" A is expected to be continuous so that the parameter update of the neural network can be smoothly realized when the gradient of the constructed loss function is backpropagated. As mentioned above, the goal of the training phase is to make the network repeatedly perform the aforementioned process to enable it to fit the perspective transformation matrix parameters between any two images with perspective transformation relationship. When used formally, the model can quickly obtain the conversion effect from different shooting angles to a unified angle.

[0066] S140 embeds the trained "vehicle recognition", "parking space recognition" and "view angle transformation" CNN model algorithms into the front-end camera monitoring device through embedded development.

[0067] S150 constructs the original digitized 2D parking lot panoramic base map: the trained model is deployed and integrated into the front-end camera device to form three modules of "vehicle recognition", "parking space recognition" and "perspective transformation". The "vehicle recognition" and "parking space recognition" modules are automatically run when the device is started, and the initial parking space information under each monitoring perspective image can be automatically obtained. Then, the "perspective transformation" module is run to project and splice the recognition results under each perspective to form the original digitized 2D parking lot panoramic base map, which is stored and shared to the management end data background.

[0068] Figure 6 An embodiment of an image sequence splicing and parking lot 2D panoramic electronic base map generation model is given, which is another key link for generating a parking lot panoramic electronic base map. Some bird's-eye view or near-vertical overhead view images of the parking lot can be collected by a drone or a high-position camera, i.e. Figure 6 the "calibration template image taken by the high-position overhead view camera" described in the above embodiment. These images are used as references, and are jointly input into the perspective transformation model described in Figure 5 , i.e. the conversion parameters between each camera perspective and the vertical overhead view can be quickly obtained. In the figure, H 1T , H 2T , H 3T , etc. represent the perspective conversion matrices between the image sequences under each monitoring perspective and the bird's-eye view or vertical overhead view image used as a reference. The inverses of these matrices are applied to a series of image sequences under different monitoring perspectives, and perspective transformation is performed on each monitoring perspective image. As shown in the layout mode in Figure 2 , there are the same objects (overlapping parts) between the images taken by adjacent cameras in these image sequences, so the converted image sequences also have overlapping parts. At this time, the image splicing and fusion algorithm interface based on feature point matching can be directly called, and methods such as APAP, SPHP, Poisson fusion, etc. can be used to splice and fuse the entire image to obtain the original 2D parking lot panoramic parking space occupancy base map.

[0069] Figure 7The present application is explained in the whole logical structure and data flow of the algorithm with specific embodiments. Details include: ①A series of image sequences collected by the front-end camera are used as the input of this link. Since the remaining scattered free parking spaces in the monitoring image at any time are very common, the vehicle recognition model and the parking space recognition model are called in turn as appropriate to detect the vehicle parking position and the free parking space under each monitoring view. ②The model assumes that there is a one-to-one correspondence between the vehicle parking position and the parking space position. Through the idea of dynamic background generation, the pixel coordinates of the vehicle parking position and the free parking space position are batched and iterated in a sliding window manner. The pixel coordinates of the center point of the anchor box output by the vehicle recognition and parking space recognition models are mainly used. The moving average method is used to calculate the weighted mean of the vehicle parking position and the free parking space position information in each iteration until the weighted mean converges. ③The converged weighted mean can represent the parking space position in a single monitoring video view. Through this value, the pixel coordinates of the center point of each parking space in the image are determined, and the boundary of each parking space is drawn with a quadrilateral frame of a specific color, thereby generating unique position recognition information for each parking space, including: the pixel coordinates of the four vertices of the bounding box corresponding to each parking space position in the image, the pixel coordinates of the center point of the bounding box, and a unique ID number. The three are in a one-to-one correspondence. The above information is superimposed and drawn to the original monitoring video image, and the resulting composite image is the "single monitoring camera imaging image with drawn parking space position information" in the claims. Stacking multiple single monitoring camera imaging images results in Figure 7 "monitoring image sequence containing determined parking space position information" in the claims.

[0070] Figure 7 The view transformation module in the claims first inputs the real-time monitoring image sequence from the front end and the calibration template image obtained by shooting from the high-altitude overhead view into the aforementioned perspective transformation module to obtain the perspective transformation matrix between each monitoring view and the bird's-eye overhead view image. Then, the monitoring image sequence containing the determined parking space position information is input into the view splicing module, the relevant multi-image splicing and fusion algorithm is called, and the original digital 2D parking lot panoramic base map is generated. The panoramic base map contains the ID number, bounding box vertex, and bounding box center point coordinate information of all parking spaces in the field.

[0071] Figure 7The parking space occupation real-time detection and visualization module in the application first calls a vehicle recognition model to detect the vehicles parked in the parking lot, and in a single camera monitoring visual angle image, a nearest neighbor algorithm is first applied to roughly estimate the parking spaces that may have been occupied, and then an IOU judgment method is applied to determine the occupied parking spaces. The relationship between the "monitoring image sequence containing the detected vehicle position information" obtained by the vehicle recognition model and the "monitoring image sequence already containing the determined parking space position information" is comprehensively analyzed, and the real-time occupation of the parking spaces in all single camera monitoring visual angle images in the parking lot is determined one by one. Then, the judgment result is fused with the information of the original digitized 2D parking lot panoramic base map to generate the visualization representation of the real-time detection result of the parking space occupation, namely the real-time digitized 2D parking lot panoramic parking space occupation base map. The principle of the nearest neighbor algorithm for judging the parking space occupation state is that for single monitoring camera imaging, the position information of the parking space contained in the "single monitoring camera imaging image with drawn parking space position information" is combined, and the center point of the real-time vehicle position bounding box detected by the vehicle detection model is searched in the image. In the pixel coordinate system, the Euclidean distance of all pending parking space ID numbers is less than a certain threshold a. These pending parking spaces are the parking spaces that may have been used.

[0072] Further, the principle of the IOU judgment method proposed in the application is that the intersection over union value of the real-time vehicle position bounding box detected by the vehicle detection model and the parking space bounding box of all pending parking space ID numbers searched by the nearest neighbor algorithm is compared one by one in the "single monitoring camera imaging image with drawn parking space position information" with parking space bounding box annotation. Once the value is greater than a certain threshold b, it can be determined that the corresponding parking space has been occupied. The specific judgment process of the real-time occupation of the parking space is that a 0-1 state indication identifier is set for each parking space ID. The rule of the identifier is that the occupied parking space indication identifier is "1", and the empty available parking space indication identifier is "0". When the IOU judgment algorithm is executed, if the number of the aforementioned pending parking space ID numbers is greater than or equal to 2 after the search by the nearest neighbor algorithm, it means that the situation of "one vehicle occupying multiple parking spaces" may occur. At this time, the intersection over union of the vehicle target bounding box detected by the vehicle recognition model and the bounding box of each parking space corresponding to the pending parking space ID number is checked one by one. Once it is greater than the threshold b, it is determined that the parking space has been occupied. After the traversal is completed, the final parking space occupation judgment result is obtained according to the occupation judgment of each parking space corresponding to the pending parking space ID number. The application can identify the situation of "one vehicle occupying multiple parking spaces" by the joint application of the nearest neighbor algorithm and the IOU judgment method.

[0073] After judging the parking space occupation condition, a list with the length of the maximum number of parking space IDs is created to store the state indication identifier values of all parking spaces, the state indication identifier values are associated with the information of the original digitized 2D parking lot panoramic base map, and the occupied parking spaces and the empty available parking spaces are marked with different color marking boxes on the original digitized 2D parking lot panoramic base map, and the obtained result is the "real-time digitized 2D parking lot panoramic parking space occupation base map". The base map is a visual representation of the real-time occupation condition of the parking spaces in the parking lot, and in the embodiment, the state of the parking spaces is represented by different colors, orange represents that the current parking space is occupied, and light blue represents that the current parking space is still in an idle state.

[0074] Up to now, the goal of real-time detection of the parking space occupation state and result representation is achieved.

[0075] The application designs a reasonable camera layout through live investigation, so that the video monitoring and sensing coverage of the parking lot is as comprehensive as possible with less camera equipment cost investment. The perception real-time performance, reliability and accuracy of the real-time available parking space resources in the parking lot are improved, which provides positive help for further utilization of limited and tight parking resources, alleviates the contradiction between supply and demand of urban parking, and provides more real and perceptual panoramic visual real-time detection results for the management judgment and decision-making process of parking lot operators and city managers.

[0076] The above embodiment is a preferred embodiment of the application, but the embodiments of the application are not limited by the above embodiment, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the application shall be equivalent replacement modes and shall be included in the protection scope of the application.

Claims

1. A panoramic detection method for parking space occupancy status, characterized in that, include: Obtain the spatial layout of the parking lot to be managed, and determine the layout scheme of the cameras required to detect parking spaces in the parking lot to be managed based on the coverage of each surveillance camera. A parking space status detection module is constructed, which includes a deep learning-based vehicle recognition model and a deep learning-based parking space recognition model, to obtain information about vehicles and parking spaces in the parking lot. A panoramic base map generation module is constructed, specifically an end-to-end perspective transformation model, which includes a perspective transformation model and a perspective stitching model, used to generate a 2D panoramic base map of the photographed object. The original digital 2D panoramic image of the parking lot is constructed by obtaining initial parking space information from various monitoring perspectives through vehicle and parking space recognition models. An end-to-end perspective transformation model is then used to stitch and fuse the image sequences containing initial parking space information from each perspective to form the original digital 2D panoramic image of the parking lot. The specific workflow is as follows: The vehicle recognition model and parking space recognition model are called in sequence to detect the vehicle parking location and available parking spaces from each monitoring perspective. Assuming a one-to-one correspondence between the vehicle parking location and the vacant parking space location, a dynamic background generation method is adopted. The pixel coordinates of the vehicle parking location and the vacant parking space location are traversed in batches using a sliding window. The weighted average of the vehicle parking location and vacant parking space location information for each single traversal is calculated using the moving average method until the weighted average converges. The converged weighted average value can represent the parking space position in a single surveillance video view. This value determines the pixel coordinates of the center point of each parking space in the image, and the boundary of each parking space is drawn with a quadrilateral frame of a set color, thereby generating unique position identification information for each parking space. The position identification information includes: the pixel coordinates of the four vertices of the bounding box corresponding to each parking space position in the image, the pixel coordinates of the center point of the bounding box, and a unique ID number. These three are in a one-to-one correspondence. The above information is superimposed on the original surveillance video image, and the resulting composite image is a "single surveillance camera image with parking space position information drawn". The single surveillance camera images with parking space position information drawn are input one by one into the end-to-end view transformation model to generate the original digital 2D parking lot panoramic base map. This base map presents the stitching result of multiple "single surveillance camera images with parking space position information drawn" from a vertical bird's-eye view. The stitching result includes the ID number of all parking spaces in the parking lot, the coordinate information of the vertices and center point of the bounding box. The nearest neighbor algorithm is used to search for all undetermined parking space IDs in the digital 2D parking lot panoramic image whose Euclidean distance in pixel coordinate system to the center point of the real-time vehicle location bounding box detected by the vehicle recognition model is less than a certain threshold α. These undetermined parking spaces are those that may have already been used. The IOU (Interval of Units) method is used to determine the occupied parking spaces. The relationship between the vehicle position bounding box identified by the vehicle recognition model and the "parking space position information" contained in the "single monitoring camera image with plotted parking space position information" is determined one by one. The real-time occupancy status of parking spaces in all the single monitoring camera images in the parking lot is then determined. The determination results are then fused with the information of the original digital 2D parking lot panoramic base map to generate a visual representation of the real-time parking space detection results—the real-time digital 2D parking lot panoramic parking space occupancy base map.

2. The panoramic detection method for parking space occupancy status according to claim 1, characterized in that, The vehicle recognition model uses a lightweight, improved YOLOv4 network as its detection framework, specifically: The input image is extracted using a backbone network, which is CSP DarkNet53. The feature fusion network extracts feature maps at three scales, and the fusion weights decrease sequentially according to the scale size. The feature map output by the feature fusion network is subjected to convolution operation to obtain three feature vectors of different lengths. The feature vectors are then decoded to obtain the target information in the input image, which includes the target type and location.

3. The panoramic detection method for parking space occupancy status according to claim 1, characterized in that, The parking space recognition model uses a lightweight SSD network as its detection framework, specifically: A backbone network, namely ResNet-50, is used to extract features from the input image. The pruned SSD network is used to fuse features to obtain target information in the input image. The pruned SSD network retains only the network structure that outputs a 10×10 feature map.

4. The panoramic detection method for parking space occupancy status according to claim 1, characterized in that, The perspective transformation model uses VGG-16 as the feature extraction backbone network during the training phase. The perspective transformation model takes two image pairs that satisfy a certain perspective transformation relationship as input and outputs the offset of four feature points between the two images. By fusing the offset with the original coordinate information of the four feature points, the coordinates of the four pairs of feature points before and after the transformation can be solved, thereby obtaining the perspective transformation between the image pairs.

5. The panoramic detection method for parking space occupancy status according to claim 1, characterized in that, The visualization representation of the real-time parking space detection results—a real-time digital 2D panoramic parking space occupancy base map—is specifically as follows: For each parking space ID, a status indicator from 0 to 1 is set. The rule is that an occupied parking space is indicated by "1", and an empty or available parking space is indicated by "0". A list with a length equal to the maximum number of parking space IDs is created to store the status indicator values ​​of all parking spaces. The status indicator values ​​are associated with the information of the original digital 2D parking lot panoramic base map. On the original digital 2D parking lot panoramic base map, occupied parking spaces and empty or available parking spaces are marked with different colored marker boxes. The result is the "real-time digital 2D parking lot panoramic parking space occupancy base map", which is a visual representation of the real-time occupancy status of parking spaces.

6. The panoramic detection method for parking space occupancy status according to claim 1, characterized in that, The viewpoint stitching model is specifically as follows: First, the sequence of images from each surveillance camera is taken as input and output as a new image sequence with a vertical bird's-eye view. Then, the new image sequence is stitched together using the SIFI feature point matching method to generate a 2D panoramic base map of the parking lot.

7. The panoramic detection method for parking space occupancy status according to claim 1, characterized in that, The IOU determination method is as follows: For each "single surveillance camera image with parking space location information drawn" marked with a parking space boundary box, the intersection-union ratio (IUU) of the real-time vehicle location boundary box detected by the vehicle detection model and the boundary boxes of all undetermined parking space IDs obtained by the nearest neighbor algorithm is compared. Once this value is greater than the set threshold b, it can be determined that the corresponding parking space has been occupied.

8. A system for implementing the panoramic detection method for parking space occupancy status according to any one of claims 1-7, characterized in that, Multiple surveillance cameras are installed above the parking lot to be managed, covering the entire parking area. Each camera has a built-in parking space status detection module and a panoramic base map generation module. Upon startup, the vehicle recognition model and parking space recognition model are automatically run to automatically acquire the initial parking space information from each monitoring perspective. Then, the panoramic base map generation module is run to stitch and merge the image sequence containing the initial parking space information from each perspective to form a panoramic 2D digital electronic base map of the parking lot parking spaces, which is then stored and shared to the management data backend.

Images