Real-time panoramic parking space detection method and device based on double-network deep learning

Abstract

The application provides a real-time panoramic parking space detection method based on double-network deep learning, which has good detection effect on diagonal parking spaces, supports parking space detection in the case where there is no parking space line or the parking space line is incomplete, and comprises the following steps: calling a trained deep learning parking space instance segmentation model, detecting real-time panoramic video frames, and obtaining segmented parking space line results; extracting feature points to fit the parking space line; logically judging and combining the obtained parking space line according to the constraints of conventional parking spaces to form parking spaces, and saving a candidate parking space result set P1 obtained through the deep learning parking space instance segmentation model; calling a trained deep learning parking space corner point detection model to obtain a parking space corner point detection result; combining the accurate angle value obtained through parking space segmentation to calculate a candidate parking space result set P2; performing comprehensive logical judgment to screen out a detection parking space set with the best confidence; and combining track prediction to track the position of the parking space detected in the historical frames to perform position compensation.

Description

Technical Field

[0001] This invention relates to the field of assisted parking technology, specifically to a real-time panoramic parking space detection method, device, and storage medium based on dual-network deep learning. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning, automatic parking, remote parking, and memory parking are becoming increasingly widely used. The advantages of deep learning are becoming more and more prominent; therefore, parking space detection based on deep learning vision is receiving more research, especially memory parking, which requires a higher recognition rate for parking space detection in order to construct a more complete map and store parking space information.

[0003] Currently, common deep learning-based panoramic parking space detection networks include deep learning networks for parking space corner detection and deep learning networks for segmenting parking space lines. Currently, deep learning networks for parking space corner detection perform well for right-angled parking spaces, but their performance is less than ideal for non-right-angled angled parking spaces. In particular, when regressing the angle of the angled parking space, the obtained angle differs from the actual scene, ultimately affecting the final parking result and preventing the vehicle from being centered. Some scenarios lack parking space corner information or have severely worn corner points, which poses a challenge for parking space corner detection. While deep learning networks that segment parking lines can effectively solve the problem of inaccurate positioning of angled parking spaces, the segmentation method requires significantly more information for parking lines or masonry in combined parking spaces than for parking space corner detection. In some special cases, insufficient parking line information leads to unsatisfactory detection results. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a real-time panoramic parking space detection method, device, and storage medium based on dual-network deep learning. It supports the detection of various parking space types, exhibits good detection performance for angled parking spaces, and supports parking space detection in cases where parking lines are absent or incomplete.

[0005] The technical solution is as follows: a real-time panoramic parking space detection method based on dual-network deep learning, characterized by the following steps: The system acquires real-time video data from cameras in the four directions (front, rear, left, and right) of the vehicle and synthesizes it into real-time panoramic video frames. The trained deep learning parking space instance segmentation model is called to detect real-time panoramic video frames and obtain the segmented parking space lines. Feature points are extracted from the segmented parking lines by traversing horizontally and vertically respectively; parking feature points at any angle are obtained. The obtained feature points are fitted along the horizontal and vertical axes to obtain parking lines at any angle. The obtained parking lines are combined into parking spaces by logical judgment according to the constraints of regular parking spaces. The regular parking spaces include parallel parking spaces, perpendicular parking spaces, diagonal right-angle parking spaces and diagonal non-right-angle parking spaces. The candidate parking space result set P1 obtained by the deep learning parking space instance segmentation model is saved. The trained deep learning parking space corner detection model is called to detect real-time panoramic video frames and obtain the parking space corner detection results. The detected parking space corner points are matched and combined with the precise angle values ​​obtained from parking space segmentation to calculate the candidate parking space result set P2; A comprehensive logical judgment is made on the candidate parking space result set P1 and candidate parking space result set P2 obtained from the detection, and finally the detection parking space set with the highest confidence is selected. The parking spaces detected in historical frames are tracked and their positions are compensated by combining trajectory estimation.

[0006] Furthermore, a deep learning parking space corner detection model is constructed based on a convolutional neural network. The deep learning parking space corner detection model is used to learn the features of parking space corners and outputs a target box containing the detected parking space corner, as well as the type and confidence of the target box. The target box is represented by the coordinates of the top left corner and the bottom right corner of the target box in the original image.

[0007] Furthermore, when constructing the training set for the deep learning parking space corner detection model, the target boxes of the parking space corners in the synthesized panoramic image are labeled and saved separately. The storage path of the original image is added to the original image list, and the storage path of the panoramic image with labeled parking space corners is added to the labeled image list. The deep learning parking space corner detection model is then called to load the original image list and the corresponding labeled image list for iterative training.

[0008] Furthermore, the labels for parking space corners include vertical T-shaped, vertical L-shaped, inclined T-shaped, inclined L-shaped, cross-shaped, U-shaped, straight, disabled parking space, no-parking space P, and no-parking space X.

[0009] Furthermore, the process involves performing a comprehensive logical judgment on the detected candidate parking space result set P1 and candidate parking space result set P2 to ultimately select the detected parking space set with the highest confidence level, as follows: If candidate parking space result sets P1 and P2 have overlapping parking spaces, then based on the parking space line numbers stored in candidate parking space result set P1, find all four parking space lines corresponding to the corner points of the parking space. Then, determine whether any of these four parking space lines passes through the two corner points of the parking space in candidate parking space result set P2. If they do, the parking space in candidate parking space result set P2 is considered to have higher reliability. Based on the included angle of the parking spaces in candidate parking space result set P1, calculate the other two corner points of the parking space in candidate parking space result set P2. Finally, output the parking spaces in candidate parking space result set P2, set the parking spaces in candidate parking space result set P2 to reliable, and set the parking spaces in candidate parking space result set P1 to unreliable. If they do not pass through, directly output the parking spaces in candidate parking space result set P1, set the parking spaces in candidate parking space result set P1 to reliable, and set the parking spaces in candidate parking space result set P2 to unreliable. If there is no overlap between the candidate parking space result set P1 and the candidate parking space result set P2, then the other two corner points of the parking spaces in the candidate parking space result set P2 in this frame are calculated based on the angle between the parking spaces on the same side in the historical frame. At the same time, the parking spaces in both the candidate parking space result set P1 and the candidate parking space result set P2 are set as reliable parking spaces and output.

[0010] Furthermore, if the coordinate difference between two corner points of candidate parking spaces in candidate parking space result set P1 and candidate parking space result set P2 is within 50cm, then candidate parking space result set P1 and candidate parking space result set P2 are considered to have overlapping parking spaces.

[0011] Furthermore, when extracting feature points from the parking line segmentation results that are traversed simultaneously in both the horizontal and vertical directions, feature points are extracted by sliding based on the local gray-scale gradient values ​​above and below the parking line. The horizontal traversal range is [0°, 45°] and [135°, 180°], and the vertical traversal range is [45°, 135°]. The traversal angle range is the angle between the horizontal line and the horizontal line.

[0012] Furthermore, a deep learning parking space instance segmentation model is constructed based on a convolutional neural network. The deep learning parking space instance segmentation model includes a Bias layer, a combination module of Convolution layer + BatchNorm layer + ReLU layer, a pooling layer, a Deconvolution layer, and an Eltwise layer. The deep learning parking space instance segmentation model first learns each pixel of the input sample through convolutional downsampling, and then outputs a panoramic segmentation result image of the same size as the original input image through deconvolution. The training set is set to train the deep learning parking space instance segmentation model until the model converges to obtain the trained deep learning parking space instance segmentation model.

[0013] Furthermore, when constructing the training set, the parking lines in the synthesized panoramic image are labeled and saved separately. The storage path of the original image is added to the original image list, and the storage path of the panoramic image with labeled parking lines is added to the segmentation label image list. The deep learning parking space instance segmentation model is then called to load the original image list and the corresponding segmentation label image list for iterative training.

[0014] Furthermore, when logically combining parking spaces according to the constraints of regular parking spaces, the condition for parallel parking space constraints is that they must be satisfied simultaneously: C1.1: The angle formed by the parking space lines is 90°; C1.2: The distance between the two corner points of the parking spaces closest to the vehicle is greater than 450cm and less than 650cm; C1.3: The line connecting the two corner points of the parking spaces closest to the vehicle should be parallel to the vehicle; The constraint condition for determining a perpendicular parking space is that the following conditions must be met simultaneously: C2.1: The angle between the parking space lines and the parking space is 90°; C2.2: The distance between the two corner points of the parking spaces closest to the vehicle is greater than 200cm and less than 350cm; C2.3: The line connecting the two corner points closest to the vehicle should be parallel to the vehicle. The constraint condition for determining a right-angled parking space is that it must simultaneously satisfy: C3.1: The angle formed by the parking space lines is 90°; C3.2: And the distance between the two corner points of the parking spaces closest to the vehicle is greater than 200cm and less than 350cm; C3.3: The angle between the line connecting the two corner points of the parking space closest to the vehicle and the vehicle should be [30°, 60°].

[0015] The constraint condition for determining whether a parking space is diagonally arranged but not at a right angle is that it simultaneously satisfies: C4.1: When the parking space lines form an angle between the parking spaces [30°, 60°]; C4.2: And the distance between the two corner points of the parking spaces closest to the vehicle is greater than 200cm and less than 350cm; C4.3: The line connecting the two corners closest to the vehicle should be parallel to the vehicle.

[0016] Furthermore, if the angle between the line connecting the two corner points of the parking space closest to the vehicle and one of the coordinate axes of the coordinate system with the vehicle center is less than 5°, it is assumed that the line connecting the two corner points of the vehicle is parallel to the vehicle.

[0017] Furthermore, combining trajectory extrapolation to track parking spaces detected in historical frames for position compensation specifically includes: Vehicle signals are input through an external interface. These signals include gear position signal, left rear wheel speed pulse LP, right rear wheel speed pulse RP, and timestamp. The compensation value is obtained by multiplying the difference between the timestamps of consecutive frames by the wheel speed pulse. The position coordinates of the parking space at this moment are determined by whether the gear position signal indicates forward or reverse gear, and the compensation value is added or subtracted, thereby synchronizing the position of the parking space in the current frame.

[0018] A computer device, characterized in that it comprises: a processor, a memory, and a program; The program is stored in the memory, and the processor calls the program stored in the memory to execute the above-described real-time panoramic parking space detection method based on dual-network deep learning.

[0019] A computer-readable storage medium is characterized in that: the computer-readable storage medium is used to store a program for executing the above-described real-time panoramic parking space detection method based on dual-network deep learning.

[0020] This invention combines deep learning for parking space corner detection, deep learning for parking space line instance segmentation, and traditional feature point post-processing for parking space extraction. It can simultaneously detect parallel parking spaces, perpendicular parking spaces, diagonal non-right-angle parking spaces, and diagonal right-angle parking spaces, and supports parking space detection in cases where there are no parking lines or incomplete parking lines. This invention integrates these two deep learning detection methods to compensate for their respective advantages and disadvantages. For parking space corner detection with high confidence, parking space corners are used to combine and match parking spaces. The precise angle of the parking space matched and combined by parking space corners is calculated from the segmented parking space lines. For parking space lines with high confidence, parking space lines are used to match and combine parking spaces. This invention uses a deep learning parking space instance segmentation model to quickly and accurately locate the position information of segmented parking spaces, and uses a deep learning parking space corner detection model to quickly locate the position information of parking space corners. If only one network model fails to detect a parking space, it does not affect the detection of parking spaces by the other network model, thereby greatly improving the overall parking space detection and recognition rate. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating the steps of a real-time panoramic parking space detection method based on deep learning with dual networks in an embodiment. Figure 2 This is a diagram illustrating the corner labels for parking spaces. Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be described in detail below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] As shown in the background art, existing deep learning parking space detection methods have defects. Deep learning networks for parking space corner detection do not perform well for non-right-angled oblique parking spaces. Deep learning networks for segmenting parking space lines require significantly more information for parking space lines or masonry in combined parking spaces than for parking space corner detection. In some special cases, the parking space line information is insufficient, resulting in unsatisfactory detection results.

[0024] This invention provides a real-time panoramic parking space detection method based on deep learning with dual networks, see [link to relevant documentation]. Figure 1 This includes the following steps: Step 1: Acquire real-time video data from cameras in the four directions (front, rear, left, and right) of the vehicle in the panoramic system, and synthesize them to obtain real-time panoramic video frames. Step 2: Call the trained deep learning parking space instance segmentation model to detect real-time panoramic video frames and obtain the segmented parking space lines; Step 3: Extract feature points from the segmented parking lines by traversing horizontally and vertically respectively; obtain parking feature points at any angle; Step 4: Fit the obtained feature points along the horizontal and vertical axes to obtain parking lines at any angle; Step 5: Combine the obtained parking space lines into parking spaces according to the constraints of regular parking spaces. Regular parking spaces include parallel parking spaces, perpendicular parking spaces, diagonal right-angle parking spaces and diagonal non-right-angle parking spaces. Save the candidate parking space result set P1 obtained by the deep learning parking space instance segmentation model. Step 6: Call the trained deep learning parking space corner detection model to detect real-time panoramic video frames and obtain the parking space corner detection results; Step 7: Match the detected parking space corner points and combine them with the precise angle values ​​obtained from parking space segmentation to calculate the candidate parking space result set P2; Step 8: Perform a comprehensive logical judgment on the candidate parking space result set P1 and candidate parking space result set P2 obtained from the detection, and finally select the detection parking space set with the highest confidence. Step 9: Combine trajectory estimation to track the parking spaces detected in historical frames and perform position compensation.

[0025] In one specific embodiment of the present invention, in step 1, fisheye camera videos in the four directions of the vehicle (front, rear, left, and right) are collected in real time, and the panoramic system is called to synthesize the images collected by the cameras to obtain real-time panoramic video frames. In step 2, a deep learning parking space instance segmentation model is constructed based on a convolutional neural network. The deep learning parking space instance segmentation model includes a Bias layer, a combination module of Convolution layer + BatchNorm layer + ReLU layer, a pooling layer, a Deconvolution layer, and an Eltwise layer. The deep learning parking space instance segmentation model first learns each pixel of the input sample through convolutional downsampling, and then outputs a panoramic segmentation result image of the same size as the original input image through deconvolution. The deep learning parking space instance segmentation model classifies each pixel of the input panoramic parking space image, where 0 represents non-parking lines, i.e., the background, and 1 represents parking lines, i.e., the segmentation target.

[0026] To train a deep learning parking space instance segmentation model, a training set is set up. During training, parking lines in the synthesized panoramic image are labeled and saved separately. The storage paths of the original images are placed in the original image list `image.txt`, which contains the storage paths of each original image. The storage paths of the panoramic images with labeled parking lines are placed in the segmentation label image list `label.txt`. The deep learning parking space instance segmentation model is then called to load the original image list and the corresponding segmentation label image list for iterative training until the model converges, resulting in a well-trained deep learning parking space instance segmentation model.

[0027] Specifically, in step 3, when extracting feature points from the parking line segmentation results that are traversed and segmented simultaneously in both the horizontal and vertical directions, feature points are extracted by sliding based on the local gray-scale gradient values ​​above and below the parking line. The horizontal traversal range is [0°, 45°] and [135°, 180°], and the vertical traversal range is [45°, 135°]. The traversal angle range is the angle between the traversal and the horizontal line, thus obtaining parking line features at any angle and realizing parking space detection at any angle, including not only right-angled parking spaces but also non-right-angled parking spaces. Specifically, in step 4, the obtained feature points are fitted in both the horizontal and vertical directions to obtain parking lines at arbitrary angles, forming feature line segments; whether a feature line segment belongs to a straight line can be determined by the slope of the straight line; feature line segments that belong to a straight line are fitted into a straight line, and the fitted straight line is used as the parking line. Specifically, in step 5, the obtained parking space lines are logically combined into parking spaces according to the constraints of regular parking spaces. Regular parking spaces include parallel parking spaces, perpendicular parking spaces, diagonal right-angle parking spaces, and diagonal non-right-angle parking spaces. When combining parking spaces according to the constraints of regular parking spaces, the condition for parallel parking space constraints is that they must be satisfied simultaneously: C1.1: The angle formed by the parking space lines is 90°; C1.2: The distance between the two corner points of the parking spaces closest to the vehicle is greater than 450cm and less than 650cm; C1.3: The line connecting the two corner points of the parking space closest to the vehicle is considered to be parallel to the vehicle when the angle between the line connecting the two corner points of the parking space closest to the vehicle and one of the coordinate axes of the coordinate system with the center of the vehicle is less than 5°. The constraint condition for determining a perpendicular parking space is that the following conditions must be met simultaneously: C2.1: The angle between the parking space lines and the parking space is 90°; C2.2: The distance between the two corner points of the parking spaces closest to the vehicle is greater than 200cm and less than 350cm; C2.3: The line connecting the two corner points closest to the vehicle is considered to be parallel to the vehicle when the angle between the line connecting the two corner points closest to the vehicle and one of the coordinate axes of the coordinate system with the vehicle center is less than 5°. The constraint condition for determining a right-angled parking space is that it must simultaneously satisfy: C3.1: The angle formed by the parking space lines is 90°; C3.2: And the distance between the two corner points of the parking spaces closest to the vehicle is greater than 200cm and less than 350cm; C3.3: The angle between the line connecting the two corner points of the parking space closest to the vehicle and the vehicle should be [30°, 60°].

[0028] The constraint condition for determining whether a parking space is diagonally arranged but not at a right angle is that it simultaneously satisfies: C4.1: When the parking space lines form an angle between the parking spaces [30°, 60°]; C4.2: And the distance between the two corner points of the parking spaces closest to the vehicle is greater than 200cm and less than 350cm; C4.3: The line connecting the two corner points closest to the vehicle is considered to be parallel to the vehicle when the angle between the line connecting the two corner points closest to the vehicle and one of the coordinate axes of the coordinate system with the vehicle center is less than 5°.

[0029] The above constraints are used to determine and combine parking spaces, and the candidate parking space result set P1 obtained by the deep learning parking space instance segmentation model is saved.

[0030] In addition to the deep learning parking space instance segmentation model in step 2 above, the embodiment also uses a deep learning parking space corner detection model to accurately obtain the parking space corner location information.

[0031] Specifically, in step 6 of the embodiment, a deep learning parking space corner detection model is constructed based on a convolutional neural network. The deep learning parking space corner detection model includes convolutional layers, batch normalization (BN) layers, ReLU activation functions, and pooling layers. The deep learning parking space corner detection model is built based on the SSD model, and data transformation is performed on the deep learning parking space corner detection model to calculate floating-point numbers as 8-bit integers, thereby improving computational efficiency and speed and achieving real-time performance. The deep learning parking space corner detection model is used to learn the features of parking space corners and outputs the target boxes containing the detected parking space corners, as well as the type and confidence of the target boxes. The target boxes are represented by the coordinates of the top left and bottom right corners of the target boxes in the original image. The confidence of the detected parking space corner target boxes is output. The confidence of the deep learning parking space corner detection model is the credibility of the detected parking space corner target boxes.

[0032] When constructing the training set for the deep learning parking space corner detection model, the target boxes of the parking space corners in the synthesized panoramic image are labeled and saved separately. The storage path of the original image is added to the original image list, and the storage path of the panoramic image with labeled parking space corners is added to the labeled image list. The deep learning parking space corner detection model is then called to load the original image list and the corresponding labeled image list for iterative training.

[0033] like Figure 2 As shown, the types of parking space corner labels in this embodiment are classified into ten categories, including vertical T-shaped, vertical L-shaped, inclined T-shaped, inclined L-shaped, cross-shaped, U-shaped, straight, disabled parking space, no-parking space P, and no-parking space X.

[0034] Specifically, in step 7 of the embodiment, the detected parking space corner points are matched, and combined with the precise angle values ​​obtained from parking space segmentation, the other two parking space corner points are calculated to obtain the candidate parking space result set P2. Specifically, in step 8 of the embodiment, a comprehensive logical judgment is performed on the detected candidate parking space result set P1 and candidate parking space result set P2 to finally select the detected parking space set with the highest confidence; the specific execution is as follows: For the parking space obtained in the candidate parking space result set P1, the parking space contains complete parking space information, including: parking space corner point, parking space type, parking space orientation, and parking space angle. These can be calculated by the information of each segmented parking space line. The parking spaces obtained in the candidate parking space result set P2 are obtained through corner point detection. The parking space information contained only includes the two corner points of the parking space. The parking space information is incomplete because only the two corner points of the parking space may be detected in the panoramic view. At this time, the included angle of the parking space cannot be accurately determined, and therefore the other two corner points of the parking space cannot be calculated.

[0035] In the embodiment, if the coordinate difference between two corner points of candidate parking spaces in candidate parking space result set P1 and candidate parking space result set P2 is within 50cm, then candidate parking space result set P1 and candidate parking space result set P2 are considered to have overlapping parking spaces.

[0036] If candidate parking space result sets P1 and P2 have overlapping parking spaces, then based on the parking space line numbers stored in candidate parking space result set P1, find all four parking space lines corresponding to the corner points of the parking space. Then, determine whether any of these four parking space lines passes through the two corner points of the parking space in candidate parking space result set P2. If they do, the parking space in candidate parking space result set P2 is considered to have higher reliability. Based on the included angle of the parking spaces in candidate parking space result set P1, calculate the other two corner points of the parking space in candidate parking space result set P2. Finally, output the parking spaces in candidate parking space result set P2, set the parking spaces in candidate parking space result set P2 to reliable, and set the parking spaces in candidate parking space result set P1 to unreliable. If they do not pass through, directly output the parking spaces in candidate parking space result set P1, set the parking spaces in candidate parking space result set P1 to reliable, and set the parking spaces in candidate parking space result set P2 to unreliable. If there is no overlap between the candidate parking space result set P1 and the candidate parking space result set P2, then the other two corner points of the parking spaces in the candidate parking space result set P2 in this frame are calculated based on the angle between the parking spaces on the same side in the historical frame. At the same time, the parking spaces in both the candidate parking space result set P1 and the candidate parking space result set P2 are set as reliable parking spaces and output.

[0037] Step 8 simultaneously considers parking space segmentation and parking space corner detection to perform comprehensive logical judgment on parking spaces in all scenarios, resulting in excellent detection results.

[0038] In step 8 of the embodiment, the vehicle position confidence includes the length L of the parking space line, the parallelism P of the opposite side of the parking space, and the calculated confidence S of the distance between the corner point of the parking space and the two nearest neighboring parking space lines. The vehicle position confidence C is calculated as L * 0.3 + P * 0.4 + S * 0.3. In step 9 of the embodiment, the location compensation for tracking the parking spaces detected in historical frames by combining trajectory estimation specifically includes: Vehicle signals are input through an external interface. These signals include gear position signal, left rear wheel speed pulse LP, right rear wheel speed pulse RP, and timestamp. The gear position signal includes forward gear D and reverse gear R. The compensation value is obtained by multiplying the difference between the timestamps of the preceding and following frames by the wheel speed pulse. Depending on whether the gear position signal indicates forward or reverse gear, the position coordinates of the parking space at that moment are determined to either add or subtract the compensation value, thereby synchronizing the position of the parking space in the current frame. Position compensation makes the detected parking space more stable and accurate.

[0039] Traditional non-deep learning methods for parking space detection extract feature points based on pixel gradient differences in the image, meaning they rely on local pixel differences in the input image. This type of traditional method is highly dependent on lighting conditions; factors such as light intensity, brightness, ground reflectivity, and differences between the pixel values ​​of the parking lines and the surrounding background all have an impact. This patent, however, utilizes deep learning. First, a deep learning parking space instance segmentation model segments the parking lines, treating the remaining areas as background. Then, feature points are extracted from the segmented parking lines. The principle is to determine whether a point is a parking line feature point based on the segmentation result: a segmentation value of 0 indicates background, and a segmentation value of 1 indicates a parking line. Next, feature points are extracted from the segmented parking lines using horizontal and vertical traversal. These feature points are then fitted horizontally and vertically to obtain parking lines at arbitrary angles, and parking spaces are determined through constraints. Compared to traditional methods… This method better addresses parking space detection in situations where parking lines are absent or incomplete. It supports parking space detection in environments with brick or stone surfaces, parking spaces with lines partially covered by weeds, parking spaces under trees, and parking spaces in low-light conditions in underground garages where detection is ineffective. It accelerates processing and reduces runtime by using traditional methods for feature point extraction after deep learning. It can simultaneously detect parallel, perpendicular, diagonally non-right-angled, and diagonally right-angled parking spaces. Compared to some other deep learning methods that use a single deep learning network to perform two tasks—segmenting parking lines and performing logistic regression localization on the segmented lines—which takes longer, this patent's deep learning parking space instance segmentation model only segments the parking lines and uses conventional algorithms to locate the parking spaces. It does not require very powerful hardware, has low hardware dependency, and low computational requirements for the vehicle's infotainment system.

[0040] This invention combines deep learning for parking space corner detection, deep learning for parking space line instance segmentation, and traditional feature point post-processing for parking space extraction. It can simultaneously detect parallel parking spaces, perpendicular parking spaces, diagonal non-right-angle parking spaces, and diagonal right-angle parking spaces, and supports parking space detection in cases where there are no parking lines or incomplete parking lines. This invention integrates two deep learning detection methods, compensating for their respective advantages and disadvantages. For parking space corner detection with high confidence, the corner points are used to combine and match parking spaces. The precise angle of the matched parking spaces is calculated from the segmented parking space lines. For the segmented parking space lines with high confidence, the parking space lines are used to match and combine parking spaces. This invention uses a deep learning parking space instance segmentation model to quickly and accurately locate the position information of segmented parking spaces, and uses a deep learning parking space corner detection model to quickly locate the position information of parking space corners. If only one network model fails to detect a parking space, it does not affect the detection of parking spaces by the other network model, thereby greatly improving the overall parking space detection and recognition rate.

[0041] Currently, the real-time panoramic parking space detection method based on dual-network deep learning of the present invention can realize real-time parking space detection on TDA4 hardware, meeting user development needs.

[0042] In an embodiment of the present invention, a computer device is also provided, comprising: a processor, a memory, and a program; The program is stored in memory, and the processor calls the program stored in memory to execute the above-mentioned real-time panoramic parking space detection method based on dual-network deep learning.

[0043] The computer device can be a terminal, and its internal structure diagram can be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a real-time panoramic parking space detection method based on dual-network deep learning. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0044] Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory stores programs, and the processor executes these programs after receiving execution instructions.

[0045] A processor can be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. The processor can also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor.

[0046] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0047] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program for executing the above-described real-time panoramic parking space detection method based on dual-network deep learning.

[0048] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, computer devices, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0049] Embodiments of the present invention are described with reference to flowchart illustrations of methods, computer apparatuses, or computer program products according to embodiments of the invention. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart.

[0050] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in the flowchart.

[0051] The above provides a detailed description of the application of the real-time panoramic parking space detection method, computer device, and computer-readable storage medium based on deep learning with dual networks provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A real-time panoramic parking space detection method based on deep learning with dual networks, characterized in that, Includes the following steps: The system acquires real-time video data from cameras in the four directions (front, rear, left, and right) of the vehicle and synthesizes it into real-time panoramic video frames. The trained deep learning parking space instance segmentation model is called to detect real-time panoramic video frames and obtain the segmented parking space lines. Feature points are extracted from the segmented parking lines by traversing horizontally and vertically respectively; parking feature points at any angle are obtained. The obtained feature points are fitted along the horizontal and vertical axes to obtain parking lines at any angle. The obtained parking lines are combined into parking spaces by logical judgment according to the constraints of regular parking spaces. The regular parking spaces include parallel parking spaces, perpendicular parking spaces, diagonal right-angle parking spaces and diagonal non-right-angle parking spaces. The candidate parking space result set P1 obtained by the deep learning parking space instance segmentation model is saved. The trained deep learning parking space corner detection model is called to detect real-time panoramic video frames and obtain the parking space corner detection results. The detected parking space corner points are matched and combined with the precise angle values ​​obtained from parking space segmentation to calculate the candidate parking space result set P2; A comprehensive logical judgment is made on the candidate parking space result set P1 and candidate parking space result set P2 obtained from the detection, and finally the detection parking space set with the highest confidence is selected. The parking spaces detected in historical frames are tracked and their positions are compensated by combining trajectory estimation.

2. The real-time panoramic parking space detection method based on deep learning with dual networks according to claim 1, characterized in that: A deep learning parking space corner detection model is constructed based on a convolutional neural network. The deep learning parking space corner detection model is used to learn the features of parking space corners and outputs a target box containing the detected parking space corner, as well as the type and confidence of the target box. The target box is represented by the coordinates of the top left corner and the bottom right corner of the target box in the original image.

3. The real-time panoramic parking space detection method based on deep learning with dual networks according to claim 2, characterized in that: When constructing the training set for the deep learning parking space corner detection model, the target bounding boxes of the parking space corners in the synthesized panoramic image are labeled and saved separately. The storage path of the original image is added to the original image list, and the storage path of the panoramic image with labeled parking space corners is added to the labeled image list. The deep learning parking space corner detection model is called to load the original image list and the corresponding labeled image list for iterative training. The labels of the parking space corners include vertical T-shaped, vertical L-shaped, inclined T-shaped, inclined L-shaped, cross-shaped, U-shaped, straight, disabled parking space, no-parking space P, and no-parking space X.

4. The real-time panoramic parking space detection method based on deep learning with dual networks according to claim 1, characterized in that, The process involves performing a comprehensive logical judgment on the detected candidate parking space result sets P1 and P2 to ultimately select the detection parking space set with the highest confidence level, as follows: If candidate parking space result sets P1 and P2 have overlapping parking spaces, then based on the parking space line numbers stored in candidate parking space result set P1, find all four parking space lines corresponding to the corner points of the parking space. Then, determine whether any of these four parking space lines passes through the two corner points of the parking space in candidate parking space result set P2. If they do, the parking space in candidate parking space result set P2 is considered to have higher reliability. Based on the included angle of the parking spaces in candidate parking space result set P1, calculate the other two corner points of the parking space in candidate parking space result set P2. Finally, output the parking spaces in candidate parking space result set P2, set the parking spaces in candidate parking space result set P2 to reliable, and set the parking spaces in candidate parking space result set P1 to unreliable. If they do not pass through, directly output the parking spaces in candidate parking space result set P1, set the parking spaces in candidate parking space result set P1 to reliable, and set the parking spaces in candidate parking space result set P2 to unreliable. If there is no overlap between the candidate parking space result set P1 and the candidate parking space result set P2, then the other two corner points of the parking spaces in the candidate parking space result set P2 in this frame are calculated based on the angle between the parking spaces on the same side in the historical frame. At the same time, the parking spaces in both the candidate parking space result set P1 and the candidate parking space result set P2 are set as reliable parking spaces and output.

5. The real-time panoramic parking space detection method based on deep learning with dual networks according to claim 4, characterized in that: If the coordinate difference between two corner points of candidate parking spaces in candidate parking space result set P1 and candidate parking space result set P2 is within 50cm, then candidate parking space result set P1 and candidate parking space result set P2 are considered to have overlapping parking spaces.

6. The real-time panoramic parking space detection method based on deep learning with dual networks according to claim 1, characterized in that: When extracting feature points from parking lines that are segmented by traversing both horizontally and vertically, feature points are extracted by sliding based on the local gray-scale gradient values ​​above and below the parking lines. The horizontal traversal range is [0°, 45°] and [135°, 180°], and the vertical traversal range is [45°, 135°]. The traversal angle range is the angle between the traversal and the horizontal line.

7. The real-time panoramic parking space detection method based on deep learning with dual networks according to claim 6, characterized in that: A deep learning parking space instance segmentation model is constructed based on a convolutional neural network. The deep learning parking space instance segmentation model includes a Bias layer, a combination module of Convolution layer + BatchNorm layer + ReLU layer, a pooling layer, a Deconvolution layer, and an Eltwise layer. The deep learning parking space instance segmentation model first learns each pixel of the input sample through convolutional downsampling, and then outputs a panoramic segmentation result map of the same size as the original input image through deconvolution. The training set is set to train the deep learning parking space instance segmentation model until the model converges to obtain the trained deep learning parking space instance segmentation model.

8. The real-time panoramic parking space detection method based on deep learning with dual networks according to claim 7, characterized in that: When constructing the training set, the parking lines in the synthesized panoramic image are labeled and saved separately. The storage path of the original image is added to the original image list, and the storage path of the panoramic image with labeled parking lines is added to the segmentation label image list. The deep learning parking instance segmentation model is called to load the original image list and the corresponding segmentation label image list for iterative training.

9. The real-time panoramic parking space detection method based on deep learning with dual networks according to claim 1, characterized in that: When combining spaces into parking spaces using logical judgments based on the constraints of regular parking spaces, the conditions for parallel parking space constraints must be met simultaneously: C1.1: The angle formed by the parking space lines is 90°; C1.2: The distance between the two corner points of the parking spaces closest to the vehicle is greater than 450cm and less than 650cm; C1.3: The line connecting the two corner points of the parking spaces closest to the vehicle should be parallel to the vehicle; The constraint condition for determining a perpendicular parking space is that the following conditions must be met simultaneously: C2.1: The angle between the parking space lines and the parking space is 90°; C2.2: The distance between the two corner points of the parking spaces closest to the vehicle is greater than 200cm and less than 350cm; C2.3: The line connecting the two corner points closest to the vehicle should be parallel to the vehicle. The constraint condition for determining a right-angled parking space is that it must simultaneously satisfy: C3.1: The angle formed by the parking space lines is 90°; C3.2: And the distance between the two corner points of the parking spaces closest to the vehicle is greater than 200cm and less than 350cm; C3.3: The angle between the line connecting the two corner points of the parking space closest to the vehicle and the vehicle should be [30°, 60°]; The constraint condition for determining whether a parking space is diagonally arranged but not at a right angle is that it simultaneously satisfies: C4.1: When the parking space lines form an angle between the parking spaces [30°, 60°]; C4.2: And the distance between the two corner points of the parking spaces closest to the vehicle is greater than 200cm and less than 350cm; C4.3: The line connecting the two corners closest to the vehicle should be parallel to the vehicle.

10. The real-time panoramic parking space detection method based on deep learning with dual networks according to claim 9, characterized in that: When the angle between the line connecting the two corner points of the parking space closest to the vehicle and one of the coordinate axes of the coordinate system with the vehicle center is less than 5°, the line connecting the two corner points of the vehicle is considered to be parallel to the vehicle.

11. The real-time panoramic parking space detection method based on deep learning with dual networks according to claim 1, characterized in that: The method of combining trajectory extrapolation to track parking spaces detected in historical frames and performing position compensation specifically includes: Vehicle signals are input through an external interface. These signals include gear position signal, left rear wheel speed pulse LP, right rear wheel speed pulse RP, and timestamp. The compensation value is obtained by multiplying the difference between the timestamps of consecutive frames by the wheel speed pulse. The position coordinates of the parking space at this moment are determined by whether the gear position signal indicates forward or reverse gear, and the compensation value is added or subtracted, thereby synchronizing the position of the parking space in the current frame.

12. A computer device, characterized in that, It includes: a processor, memory, and programs; The program is stored in the memory, and the processor calls the program stored in the memory to execute the real-time panoramic parking space detection method based on deep learning with dual networks as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a program for executing the real-time panoramic parking space detection method based on deep learning with dual networks as described in any one of claims 1 to 11.

Images