Parking space detection method, device, electronic device and storage medium based on multi-corner point clustering

Abstract

The present invention provides a parking space detection method, device, electronic device, and storage medium based on multi-corner point clustering. By extracting heat maps and label values ​​from images at different corner point orientations, the method can cluster corner points to detect parking spaces. Based on the present invention, horizontal, vertical, and diagonal parking spaces at arbitrary angles can be accurately identified, achieving highly reliable, high-precision, and intelligent parking space detection.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent driving and automatic parking, and more specifically, to a parking space detection method, device, electronic device and storage medium based on multi-corner point clustering. Background Art

[0002] Parking space detection technology is one of the key technologies of the vehicle automatic parking system. It is the prerequisite for accurately planning the parking path and accurately parking in the parking space.

[0003] Parking space detection technology relies entirely on camera sensors to detect the specific location of parking spaces using machine vision algorithms. In recent years, this has mainly been based on traditional graphics methods, which require manual design of parking space scenarios and feature extraction. However, it is difficult to design a universal solution to address all scenarios. For example, the low detection rate in complex scenarios (such as unclear lines and grass-brick parking spaces) is currently a major pain point in visual parking space detection. Summary of the Invention

[0004] In view of this, in order to solve the above problems, the present invention provides a parking space detection method, device, electronic device and storage medium based on multi-corner clustering. The technical solution is as follows:

[0005] In one aspect, the present invention provides a parking space detection method based on multi-corner point clustering, the method comprising:

[0006] Obtaining the target image to be detected captured by the vehicle-mounted camera;

[0007] Extracting multiple heat maps and label maps of the target image, where each heat map corresponds to a corner point direction, the heat map contains heat values ​​of different pixels belonging to the corner point in the corresponding corner point direction, and the label map contains label values ​​of parking spaces to which different pixels belong;

[0008] Extracting multiple target pixels whose heat values ​​meet corresponding thresholds from each heat map, and obtaining a label value for each target pixel based on the label map;

[0009] Multiple target pixel points whose label values ​​meet the corresponding difference conditions are clustered into corner points of the same parking space in different corner point directions, and the parking space is determined based on the clustered corner points.

[0010] Optionally, extracting multiple heat maps and label maps of the target image includes:

[0011] Retrieving a pre-trained detection model, wherein the detection model is obtained by training a neural network with the goal of predicting a heat map of a sample image close to a heat map annotation result of the sample image and predicting a label map of the sample image close to a label map annotation result of the sample image;

[0012] The target image is input into the detection model to obtain multiple heat maps and label maps output by the detection model for the target image.

[0013] Optionally, the method for obtaining the sample image includes:

[0014] Acquire the first daytime sample image;

[0015] A conversion model pre-trained based on generative adversarial network technology is retrieved to convert the first daytime sample image into a first nighttime sample image through the conversion model. The conversion model is obtained by training the generative adversarial network with the goal of making the conversion result of the second daytime sample image close to the second nighttime sample image labeled by the second daytime sample image.

[0016] Optionally, the generative adversarial network includes a first generative network, a first discriminative network corresponding to the first generative network, a second generative network, and a second discriminative network corresponding to the second generative network;

[0017] Accordingly, the process of training the conversion model based on the generative adversarial network technology includes:

[0018] Inputting the second daytime sample image into the first generation network and the second identification network respectively, and inputting the second nighttime sample image into the first identification network and the second generation network, so as to achieve:

[0019] The first generation network generates a third nighttime sample image based on the second daytime sample image, and outputs the image to the first identification network, so that the first identification network processes the second nighttime sample image and the third nighttime sample image; the second generation network generates a third daytime sample image based on the second nighttime sample image, and outputs the image to the second identification network, so that the second identification network processes the third daytime sample image and the second daytime sample image;

[0020] Calculating a first loss function value based on a processing result of the first identification network, and calculating a second loss function value based on a processing result of the second identification network;

[0021] Adjust the network parameters of the first generation network, the second generation network, the first identification network, and the second identification network respectively according to the first loss function value and the second loss function value;

[0022] When the corresponding training end condition is currently met, the adjusted first generation network is used as the conversion model.

[0023] Optionally, the method further includes:

[0024] Inputting the third nighttime sample image into the second generation network, and inputting the third daytime sample image into the first generation network, to achieve:

[0025] The second generation network generates a fourth daytime sample image based on the third nighttime sample image; the first generation network generates the fourth nighttime sample image based on the third daytime sample image;

[0026] Calculating a third loss function value according to a comparison result between the second daytime sample image and the fourth daytime sample image, and a comparison result between the second nighttime sample image and the fourth nighttime sample image;

[0027] Accordingly, adjusting the network parameters of the first generation network, the second generation network, the first identification network, and the second identification network respectively according to the first loss function value and the second loss function value includes:

[0028] The network parameters of the first generation network, the second generation network, the first identification network and the second identification network are adjusted respectively according to the first loss function value, the second loss function value and the third loss function value.

[0029] Optionally, determining a parking space according to the clustered corner points includes:

[0030] Based on the parallelogram geometric features, the geometric positions of the clustered corner points are corrected to determine the parking spaces.

[0031] Another aspect of the present invention provides a parking space detection device based on multi-corner point clustering, the device comprising:

[0032] An image acquisition module is used to acquire the target image to be detected captured by the vehicle-mounted camera;

[0033] A map extraction module is configured to extract multiple heat maps and label maps from the target image, wherein each heat map corresponds to a corner point direction, the heat map contains heat values ​​indicating that different pixels belong to the corner point in the corresponding corner point direction, and the label map contains label values ​​indicating that different pixels belong to parking spaces;

[0034] The parking space detection module is used to extract multiple target pixel points whose heat values ​​meet the corresponding threshold from each heat map, and obtain the label value of each target pixel point based on the label map; cluster the multiple target pixel points whose label values ​​meet the corresponding difference conditions into corner points of the same parking space in different corner point directions, and determine the parking space based on the clustered corner points.

[0035] Optionally, the graph extraction module is specifically configured to:

[0036] Recall a pre-trained detection model, where the detection model is obtained by training a neural network with the goal of making the heat map prediction result of the sample image close to the heat map annotation result of the sample image and the label map prediction result of the sample image close to the label map annotation result of the sample image; input the target image into the detection model to obtain multiple heat maps and label maps output by the detection model for the target image.

[0037] Another aspect of the present invention provides an electronic device, comprising: at least one memory and at least one processor; the memory stores a program, the processor calls the program stored in the memory, and the program is used to implement any one of the multi-corner point clustering parking space detection methods.

[0038] Another aspect of the present invention provides a storage medium, characterized in that the storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute any one of the parking space detection methods based on multi-corner point clustering.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention provides a parking space detection method, device, electronic device, and storage medium based on multi-corner clustering. By extracting heat maps and label maps from images at different corner orientations, the method can cluster corner points to detect parking spaces. Based on the present invention, horizontal, vertical, and diagonal parking spaces at arbitrary angles can be accurately identified, achieving highly reliable, high-precision, and intelligent parking space detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0042] Figure 1 A flowchart of a parking space detection method based on multi-corner clustering provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of corner point marking provided by an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of a neural network prediction provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of the structure of a generative adversarial network provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the structure of a parking space detection device based on multi-corner point clustering provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] Parking space detection is one of the key technologies in automatic parking systems. It is the prerequisite for accurately planning parking paths and accurately parking in parking spaces. Parking space detection technology can usually be implemented using ultrasonic radar, cameras, lidar, GPS and other sensor devices, including:

[0050] 1) Ultrasonic radar: This sensor has been the primary reliance of automated parking systems in recent years. However, it can only identify parking spaces defined by vehicles in front and behind (left and right), and is unable to detect spaces with clear parking space lines but no vehicles in front or behind (left and right). While radar can accurately detect distance, the fan-shaped nature of radar waves prevents it from accurately identifying the shape of obstacles. Currently, manually set compensation values ​​are used to compensate for the coordinates of the front and rear obstacle edges. The error in the parking space edge is around 15cm, significantly impacting parking space accuracy, parking path planning, and parking performance.

[0051] 2) LiDAR: It is expensive and currently cannot be used in large quantities;

[0052] 3) Civilian GPS: The signal accuracy is poor and cannot meet the requirements for parking space identification accuracy. It cannot be used in underground parking lots and other places.

[0053] 4) Camera: The biggest advantage of machine vision is its ability to recognize object shapes. Currently, parking spaces in regular parking lots are marked with parking lines. The first three sensors cannot detect these markings; only cameras can detect line parking spaces.

[0054] Parking space detection technology relies entirely on camera sensors, using machine vision to determine the specific location of parking spaces. In recent years, visual parking space detection methods have primarily been based on traditional graphics methods. These methods require manual design of parking scenarios and feature extraction, but a universal solution is difficult to design for all scenarios. For example, the detection rate is low in complex scenarios (such as those with unclear lines or grass-covered parking spaces), which is a major pain point in visual parking space detection.

[0055] Although some novel deep learning vision methods can detect and extract parking space corner point features, with so many parking space corner points in an image, which corner points can constitute a parking space needs to be calculated in combination with parking space line features. However, parking space lines are often damaged, and relying on parking space lines to cluster parking space corner points is not reliable. Most previous methods use distance as the distance threshold. If two parking space corner points are within a certain distance range, they belong to the same parking space. This method has very high requirements on the accuracy and detection rate of corner point detection. In practice, the middle corner point of two consecutive vertical parking spaces may be missed and be considered as one horizontal parking space; there may be a false detection point in the middle of a horizontal parking space, which may be mistakenly detected as two vertical parking spaces. Therefore, accurately clustering these discrete corner points is the key point in the parking space detection method.

[0056] Therefore, to address the problems of poor performance of current parking space corner matching methods and depth-dependent corner detection performance, this paper proposes an end-to-end parking space detection solution based on a deep learning corner clustering algorithm. Compared with traditional methods, it has the following advantages:

[0057] (1) Clustering is performed on the four corner points of parking spaces. If one corner point is missing or misdetected in the detection result, it can still be corrected by using the additional three corner points, thereby improving the reliability of parking space detection in complex scenarios.

[0058] (2) Can detect oblique parking spaces.

[0059] See also Figure 1 As shown in the flowchart of the method, an embodiment of the present invention provides a parking space detection method based on multi-corner point clustering, which includes the following steps:

[0060] S10, obtaining an image of a target to be detected captured by a vehicle-mounted camera.

[0061] In an embodiment of the present invention, taking the vehicle-mounted camera as a fisheye camera as an example, the fisheye images collected by the four fisheye cameras in front, behind, left and right of the vehicle can be spliced ​​to obtain a bird's-eye view centered on the vehicle, which is used as the target image for subsequent parking space detection.

[0062] S20, extracting multiple heat maps and label maps of the target image, where one heat map corresponds to a corner point direction, the heat map contains heat values ​​of different pixels belonging to the corner point in the corresponding corner point direction, and the label map contains label values ​​of parking spaces to which different pixels belong.

[0063] In this embodiment of the present invention, a deep learning neural network can be used to perform end-to-end detection on the target image to obtain multiple heat maps and label maps of the target image. In addition, the number of heat maps extracted is at most four, because a parking space has corner points in four directions.

[0064] It should be noted that, in the embodiment of the present invention, all corner points are inner corner points, that is, one parking space has four independently corresponding inner corner points, and no two parking spaces have a common inner corner point.

[0065] In the specific implementation process, step S20 "extracting multiple heat maps and label maps of the target image" can be performed using the following steps:

[0066] Retrieve a pre-trained detection model. The detection model is obtained by training a neural network with the goal of making the heat map prediction result of the sample image close to the heat map annotation result of the sample image and the label map prediction result of the sample image close to the label map annotation result of the sample image;

[0067] Input the target image into the detection model to obtain multiple heat maps and label maps output by the detection model for the target image.

[0068] In the embodiment of the present invention, when marking corner points of sample images, they can be marked in a counterclockwise or clockwise order, and the marking order used for all sample images is the same. Figure 2 The corner point marking diagram is shown. Regardless of the relative position of the vehicle and the parking space, the corner points of the parking space are marked in a counterclockwise direction. When searching counterclockwise, the first entrance corner point encountered is corner point 1. Continuing counterclockwise, the second corner point encountered is corner point 2, followed by corner point 3, and then corner point 4.

[0069] Furthermore, the embodiment of the present invention is divided into two aspects to train the neural network:

[0070] 1) For the labeled sample images, the parking space corners are defined as hot spots using Gaussian distribution. The heat value (HeatValue) Gaussian distribution is calculated as follows:

[0071]

[0072] Among them, HeatValue is the heat value of a pixel point. The larger the value, the greater the possibility that the pixel point is a corner point. σ is a hyperparameter, representing the pixel distance factor. x0, y0 are the coordinates of the marked point (the point marked as the corner point), which can be a sub-pixel point. x, y are points within the range of σ near the marked point.

[0073] In addition, in an embodiment of the present invention, when calculating the heat value, a distance of 3*σ pixels will be extended in the four directions of up, down, left and right of the marked point to create a rectangular region of interest (if an image boundary is encountered, it will be truncated at the boundary). All points within the rectangular area participate in the calculation of the heat value. The closer the distance to the marked point, the greater the possibility that the representative is the marked point, and the larger the calculated HeatValue. Similarly, the farther the distance to the marked point, the less likely it is the marked point, and the smaller the calculated HeatValue.

[0074] The four corner points of the parking space correspond to four deep learning calculation channels. The neural network regresses the corner points marked in the sample image and uses the mean square error (MSE) loss function for model supervision. The MSE loss function is expressed as:

[0075]

[0076] Among them, m is the number of pixels involved in calculating MSE, y i is the true value, f(x i ) is the predicted value of the neural network.

[0077] For example, for a sample image, the ground-truth heatmap is generated on a 104*104*4 matrix. "4" represents four channels, representing the heatmaps for corner points 1, 2, 3, and 4, respectively. The corner points of parking spaces in the sample image are defined as hotspots using a Gaussian distribution. A neural network is used to regress the ground-truth results, using the mean square error (MSE) loss function for model supervision.

[0078] 2) For the labeled sample images, the corner points belonging to the same parking space are labeled with similar or identical label values. The label values ​​for different parking spaces vary significantly, allowing us to distinguish which four corner points belong to the same parking space. For this reason, the corner points in this embodiment of the present invention must be interior corner points. If a corner point is set as a corner point shared by two parking spaces, it will belong to two parking spaces. Therefore, it is impossible to label the corner point with a unique label value, and this solution is obviously unfeasible.

[0079] In addition, the embodiment of the present invention designs a neural network loss function that can be applied to parking space corner point clustering to supervise the training process of the neural network, so that the corner points of the same parking space obtain similar label values, and the label values ​​of different parking spaces have large differences. The loss function expression of corner point clustering is:

[0080]

[0081]

[0082] Among them, C is the number of parking spaces in the figure, N c is the number of pixels belonging to the same parking space involved in the calculation, is the mean of the label values ​​of parking space A, is the mean of the label values ​​of parking spaces B, c A For parking space A, c B It is parking space B.

[0083] It should be noted that the essence of corner clustering is that the label values ​​of the corners of the same parking space are getting closer and closer, while the label values ​​of the corners of different parking spaces are getting more and more different. Therefore, the loss functions of corner clustering are as shown in the above formula (3) and formula (4).

[0084] For example, for a sample image, a label value is generated on a 104*104*1 matrix, where the value represents the category label of each pixel parking space.

[0085] In addition, in the embodiment of the present invention, a difference threshold M of the label values ​​may be set in advance, and pixels whose label value difference is less than M are assigned to the same label.

[0086] See also Figure 3 The neural network prediction diagram is shown. The image is input into the neural network, which has two outputs: one output is a heat map of four channels, and the other output is a label map. The embodiment of the present invention does not limit the network structure of the neural network.

[0087] It should be noted that for the sample image, which is an RGB image with a size of 416*416*3, to improve the robustness of the algorithm, the embodiment of the present invention can perform random data augmentation operations on the sample image in six aspects: translation, rotation, hue, saturation, brightness, and contrast. The sample image is input into the neural network to obtain a dual-path output.

[0088] In addition, deep learning technology has made great progress in the field of visual perception in recent years. Through data-driven algorithm models, neural networks can learn a variety of visual environment features under data training to provide the algorithm with robustness in various scenarios, greatly improving the performance of visual parking space detection.

[0089] Data-driven algorithms rely on vast amounts of data as their foundation, providing diverse scenarios for optimal performance. Therefore, before training deep learning neural networks, traditional methods perform data augmentation. These methods involve image translation, rotation, scaling, modifying image contrast, saturation, and brightness, and adding random noise to ensure image diversity. However, these data augmentation operations fail to account for the effects of varying natural lighting conditions and cannot realistically simulate changes in scene lighting.

[0090] Parking space detection data collection is typically performed using dedicated data collection vehicles in various parks, residential communities, basements, and other locations with parking spaces. The time of day is the most significant factor, and since data collection mostly occurs during the day, the majority of the data in ground scenes is collected during the day. Network model predictions are effective during the day, but less so at night. Because outdoor lighting differs significantly between day and night, the images captured by the cameras also differ significantly. For deep learning neural networks, the more diverse the training dataset, the better. However, collecting data from the same community at different time periods can more than double the workload and image acquisition time, significantly impacting project development time.

[0091] Therefore, the current datasets for parking space detection algorithms based on surround view mosaics generally have a high prevalence of daytime scenes during data collection, resulting in poor results for nighttime scenes. If data collection is relied upon entirely, this will increase data collection and annotation costs, and prolong the project development cycle. To address this issue, the present invention can transform the daytime data in the dataset into nighttime scenes with and without streetlights. This not only changes the image brightness, but also the overall effect of light from vehicle lights and streetlights on the image, generating nighttime images that are imperceptible to the human eye. This method expands the dataset scenarios based on daytime data, reduces data collection costs, shortens the development cycle, and improves development efficiency.

[0092] The following describes a method for obtaining a sample image according to an embodiment of the present invention, which includes the following steps:

[0093] Acquire the first daytime sample image;

[0094] A conversion model pre-trained based on generative adversarial network technology is retrieved to convert the first daytime sample image into a first nighttime sample image through the conversion model. The conversion model is obtained by training the generative adversarial network with the goal of making the conversion result of the second daytime sample image close to the second nighttime sample image labeled by the second daytime sample image.

[0095] In an embodiment of the present invention, a generative adversarial network is trained using a second daytime sample image and a second nighttime sample image to generate a conversion model. For scenes requiring no street lights at night, the second nighttime sample image is a nighttime sample image when there are no street lights at night; and for scenes requiring street lights at night, the second nighttime sample image is a nighttime sample image when there are street lights at night. The number of second nighttime sample images in the two scenes is greater than 1000, and the number of second nighttime sample images between the two scenes is consistent. Of course, a conversion model can be trained on two scenes to convert the same daytime sample image into nighttime sample images in two scenes. The embodiment of the present invention uses one scene to illustrate the training process of the conversion model, and the same is true for the other scene.

[0096] See also Figure 4 The following is a schematic diagram of the structure of a generative adversarial network. The generative adversarial network consists of two types of network structures: a generator network and a discriminator network. The generator network uses the input image to generate an image in the target style, while the discriminator network determines whether the image output by the generator network is similar to the target style.

[0097] Continue to see Figure 4 In the embodiment of the present invention, the generative adversarial network includes a first generative network G A , and the first generation network G A The corresponding first authentication network D B , the second generation network G B And the second identification network D corresponding to the second generation network A Accordingly, the process of training the conversion model based on the generative adversarial network technology includes:

[0098] The second daytime sample image is input into the first generation network and the second identification network respectively, and the second nighttime sample image is input into the first identification network and the second generation network, so as to achieve:

[0099] The first generation network generates a third nighttime sample image based on the second daytime sample image, and outputs the image to the first identification network, so that the first identification network processes the second nighttime sample image and the third nighttime sample image; the second generation network generates a third daytime sample image based on the second nighttime sample image, and outputs the image to the second identification network, so that the second identification network processes the third daytime sample image and the second daytime sample image;

[0100] Calculating a first loss function value based on a processing result of the first identification network, and calculating a second loss function value based on a processing result of the second identification network;

[0101] Adjust the network parameters of the first generation network, the second generation network, the first identification network, and the second identification network respectively according to the first loss function value and the second loss function value;

[0102] When the corresponding training end condition is currently met, the adjusted first generation network is used as the conversion model.

[0103] Assume that the second daytime sample image is X, and the second nighttime sample image is an image of a scene without street lights at night, which is Y. To generate nighttime data using daytime parking data, X is input into the first generation network G A In the first generation network G A The output is G A (X), it is necessary to determine the first generation network G A The output image G A (X) Whether it meets the scene requirements needs to be input into the first identification network D B In the first identification network D B Processing of input Y and G A (X) Complete the authentication, the first authentication network D B The identification indicator is the following adversarial loss function:

[0104]

[0105] In addition, Y is input to the second generation network G B In the second generation network G B The output is G B (Y), it is necessary to determine the second generation network G B The output image G B (Y) Whether it meets the scene requirements needs to be input into the second identification network D A In the second identification network D A Processing the input X and G B (Y) Complete the authentication, the second authentication network D A The identification indicator is the following adversarial loss function:

[0106]

[0107] Therefore, the first loss function value can be calculated according to the above formula (5), and the second loss function value can be calculated according to the above formula (6). In the embodiment of the present invention, the sum of the first loss function value and the second loss function value is used as the loss function value of the generative adversarial network, and then the gradient descent algorithm is used to converge the loss function value of the generative adversarial network to adjust the network parameters of each network structure in the generative adversarial network.

[0108] In addition, the training end condition can be set for the training duration or the number of training times, which is not limited in the embodiment of the present invention.

[0109] Furthermore, in order to improve the performance of the conversion model, the embodiments of the present invention further consider:

[0110] If daytime data can generate nighttime data, and nighttime data can also generate daytime data, theoretically:

[0111] G B (G A (x))≈x (7)

[0112] G A (G B (y))≈y (8)

[0113] To this end, the present invention adds cyclic Loss to assist neural network training:

[0114]

[0115] Therefore, in this embodiment of the present invention, the total loss of the generative adversarial network is:

[0116] LOSS=Loss GAN (G A , D A ,X,Y)+Loss GAN (G B , D B ,X,Y)+γ*Loss cyc (G A , G B ) (10)

[0118] Among them, γ is the preset weight coefficient.

[0119] Based on this, the embodiment of the present invention further includes the following steps:

[0120] Input the third nighttime sample image to the second generation network, and input the third daytime sample image to the first generation network to achieve:

[0121] The second generation network generates a fourth daytime sample image based on the third nighttime sample image; the first generation network generates a fourth nighttime sample image based on the third daytime sample image;

[0122] A third loss function value is calculated according to a comparison result between the second daytime sample image and the fourth daytime sample image, and a comparison result between the second nighttime sample image and the fourth nighttime sample image.

[0123] Accordingly, the network parameters of the first generation network, the second generation network, the first identification network, and the second identification network are adjusted according to the first loss function value and the second loss function value, respectively, and the following steps can be adopted:

[0124] The network parameters of the first generation network, the second generation network, the first identification network and the second identification network are adjusted respectively according to the first loss function value, the second loss function value and the third loss function value.

[0125] In an embodiment of the present invention, the third loss function value is calculated according to the above formula (9), and then the sum of the first loss function value, the second loss function value and the third loss function value is used as the loss function value of the generative adversarial network according to the above formula (10), and then the gradient descent algorithm is used to converge the loss function value of the generative adversarial network to adjust the network parameters of each network structure in the generative adversarial network.

[0126] S30, extracting multiple target pixels whose heat values ​​meet corresponding thresholds from each heat map, and obtaining a label value for each target pixel based on the label map.

[0127] In the embodiment of the present invention, continue to refer to Figure 3 The neural network prediction diagram is shown. For the four heat maps output by the detection model, target pixels with heat values ​​greater than the corresponding threshold are extracted from each heat map in turn. Taking a heat map as an example, the heat values ​​of different pixels in the heat map can be sorted from small to large, and the pixels corresponding to the largest K heat values ​​are used as target pixels. Based on this, for a parking space in the environment where the vehicle is located, its four corner points correspond to four heat maps, and each corner point can be represented by a target pixel in the corresponding heat map. Among them, K is the maximum number of detected parking spaces.

[0128] For each target pixel in each heat map, its corresponding label value can be obtained in the label map to determine the parking space to which it belongs.

[0129] S40 , clustering a plurality of target pixel points whose label values ​​satisfy corresponding difference conditions into corner points of the same parking space in different corner point directions, and determining the parking space according to the clustered corner points.

[0130] In this embodiment of the present invention, for any two of the four heat maps, for example, the first heat map and the second heat map, if the label values ​​of a target pixel in the first heat map and a target pixel in the second heat map meet a difference condition, such as a difference value less than a predetermined difference threshold, then the two target pixels belong to the same parking space. Multiple target pixels belonging to the same parking space are then clustered to locate the four corner points of the parking space, thereby determining the parking space.

[0131] Ideally, four target pixels whose label values ​​meet the corresponding difference conditions in the four heat maps are extracted as parking space corner points to determine the specific location of the corresponding parking space. In practical applications, the confidence level of the target pixels can also be considered. Target pixels with heat values ​​below a preset minimum heat threshold are eliminated. The remaining target pixels are then geometrically corrected based on the parallelogram geometry to determine the parking space.

[0132] In summary, the present invention uses a deep learning neural network to perform end-to-end detection of parking spaces, which can accurately identify horizontal, vertical, and diagonal parking spaces at any angle, thereby obtaining a highly reliable, high-precision, and intelligent parking space detection system.

[0133] Based on the parking space detection method based on multi-corner clustering provided in the above embodiment, an embodiment of the present invention provides a device for executing the parking space detection method based on multi-corner clustering. The structural diagram of the device is shown in FIG. Figure 5 Shown, including:

[0134] The image acquisition module 10 is used to acquire the target image to be detected captured by the vehicle-mounted camera;

[0135] A map extraction module 20 is used to extract multiple heat maps and label maps from the target image. Each heat map corresponds to a corner point direction. The heat map contains the heat values ​​of different pixels belonging to the corner point in the corresponding corner point direction. The label map contains the label values ​​of the parking spaces to which different pixels belong.

[0136] The parking space detection module 30 is used to extract multiple target pixel points whose heat values ​​meet the corresponding threshold from each heat map, and obtain the label value of each target pixel point based on the label map; cluster the multiple target pixel points whose label values ​​meet the corresponding difference conditions into corner points of the same parking space in different corner point directions, and determine the parking space based on the clustered corner points.

[0137] Optionally, the graph extraction module 20 is specifically configured to:

[0138] Recall the pre-trained detection model, which is obtained by training the neural network with the goal of making the heat map prediction result of the sample image close to the heat map annotation result of the sample image and the label map prediction result of the sample image close to the label map annotation result of the sample image; input the target image into the detection model to obtain multiple heat maps and label maps output by the detection model for the target image.

[0139] Optionally, the image extraction module 20 acquires the sample image by:

[0140] Acquire a first daytime sample image; retrieve a conversion model pre-trained based on generative adversarial network technology to convert the first daytime sample image into a first nighttime sample image through the conversion model, wherein the conversion model is obtained by training the generative adversarial network with the goal of making the conversion result of the second daytime sample image close to the second nighttime sample image labeled with the second daytime sample image.

[0141] Optionally, the generative adversarial network includes a first generative network, a first discriminative network corresponding to the first generative network, a second generative network, and a second discriminative network corresponding to the second generative network;

[0142] Accordingly, the graph extraction module 20 trains the conversion model based on the generative adversarial network technology, including:

[0143] The second daytime sample image is input into the first generation network and the second identification network respectively, and the second nighttime sample image is input into the first identification network and the second generation network, so as to achieve: the first generation network generates a third nighttime sample image based on the second daytime sample image, and outputs it to the first identification network, so that the first identification network processes the second nighttime sample image and the third nighttime sample image; the second generation network generates a third daytime sample image based on the second nighttime sample image, and outputs it to the second identification network, so that the second identification network processes the third daytime sample image and the second daytime sample image; the first loss function value is calculated according to the processing result of the first identification network, and the second loss function value is calculated according to the processing result of the second identification network; the network parameters of the first generation network, the second generation network, the first identification network and the second identification network are adjusted according to the first loss function value and the second loss function value; when the corresponding training end condition is currently met, the adjusted first generation network is used as the conversion model.

[0144] Optionally, the graph extraction module 20 is further configured to:

[0145] Inputting the third nighttime sample image into the second generation network, and inputting the third daytime sample image into the first generation network, so as to achieve: the second generation network generating a fourth daytime sample image based on the third nighttime sample image; and the first generation network generating a fourth nighttime sample image based on the third daytime sample image; and calculating a third loss function value based on a comparison result between the second daytime sample image and the fourth daytime sample image, and a comparison result between the second nighttime sample image and the fourth nighttime sample image;

[0146] Accordingly, the graph extraction module 20 adjusts the network parameters of the first generation network, the second generation network, the first identification network, and the second identification network according to the first loss function value and the second loss function value, respectively, including:

[0147] The network parameters of the first generation network, the second generation network, the first identification network and the second identification network are adjusted respectively according to the first loss function value, the second loss function value and the third loss function value.

[0148] Optionally, the process of the parking space detection module 30 determining the parking space according to the clustered corner points includes:

[0149] Based on the parallelogram geometric features, the geometric positions of the clustered corner points are corrected to determine the parking spaces.

[0150] Based on the parking space detection method based on multi-corner point clustering provided in the above embodiment, an embodiment of the present invention also provides an electronic device, which includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, and the program is used to implement the parking space detection method based on multi-corner point clustering.

[0151] Based on the parking space detection method based on multi-corner point clustering provided in the above embodiment, an embodiment of the present invention further provides a storage medium storing computer-executable instructions for executing the parking space detection method based on multi-corner point clustering.

[0152] The above is a detailed introduction to the parking space detection method, device, electronic device and storage medium for multi-corner point clustering provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0153] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0154] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that the process, method, article, or apparatus comprising a series of elements inherent to the elements, or also including elements inherent to these processes, methods, articles, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0155] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A parking space detection method based on multi-corner clustering, characterized in that: The method comprises: Obtaining the target image to be detected captured by the vehicle-mounted camera; Extracting multiple heat maps and label maps of the target image, where each heat map corresponds to a corner point direction, the heat map contains heat values ​​of different pixels belonging to the corner point in the corresponding corner point direction, and the label map contains label values ​​of parking spaces to which different pixels belong; Extracting multiple target pixels whose heat values ​​meet corresponding thresholds from each heat map, and obtaining a label value for each target pixel based on the label map; Multiple target pixel points whose label values ​​meet the corresponding difference conditions are clustered into corner points of the same parking space in different corner point directions, and the parking space is determined based on the clustered corner points.

2. The method according to claim 1, characterized in that The extracting of multiple heat maps and label maps of the target image includes: Retrieving a pre-trained detection model, wherein the detection model is obtained by training a neural network with the goal of predicting a heat map of a sample image close to a heat map annotation result of the sample image and predicting a label map of the sample image close to a label map annotation result of the sample image; The target image is input into the detection model to obtain multiple heat maps and label maps output by the detection model for the target image.

3. The method according to claim 2, characterized in that The method for obtaining the sample image includes: Acquire the first daytime sample image; A conversion model pre-trained based on generative adversarial network technology is retrieved to convert the first daytime sample image into a first nighttime sample image through the conversion model. The conversion model is obtained by training the generative adversarial network with the goal of making the conversion result of the second daytime sample image close to the second nighttime sample image labeled by the second daytime sample image.

4. The method according to claim 3, characterized in that The generative adversarial network includes a first generative network, a first discriminative network corresponding to the first generative network, a second generative network, and a second discriminative network corresponding to the second generative network; Accordingly, the process of training the conversion model based on the generative adversarial network technology includes: Inputting the second daytime sample image into the first generation network and the second identification network respectively, and inputting the second nighttime sample image into the first identification network and the second generation network, so as to achieve: The first generation network generates a third nighttime sample image based on the second daytime sample image, and outputs the image to the first identification network, so that the first identification network processes the second nighttime sample image and the third nighttime sample image; the second generation network generates a third daytime sample image based on the second nighttime sample image, and outputs the image to the second identification network, so that the second identification network processes the third daytime sample image and the second daytime sample image; Calculating a first loss function value based on a processing result of the first identification network, and calculating a second loss function value based on a processing result of the second identification network; Adjust the network parameters of the first generation network, the second generation network, the first identification network, and the second identification network respectively according to the first loss function value and the second loss function value; When the corresponding training end condition is currently met, the adjusted first generation network is used as the conversion model.

5. The method according to claim 4, characterized in that The method further comprises: Inputting the third nighttime sample image into the second generation network, and inputting the third daytime sample image into the first generation network, to achieve: The second generation network generates a fourth daytime sample image based on the third nighttime sample image; the first generation network generates a fourth nighttime sample image based on the third daytime sample image; Calculating a third loss function value according to a comparison result between the second daytime sample image and the fourth daytime sample image, and a comparison result between the second nighttime sample image and the fourth nighttime sample image; Accordingly, adjusting the network parameters of the first generation network, the second generation network, the first identification network, and the second identification network respectively according to the first loss function value and the second loss function value includes: The network parameters of the first generation network, the second generation network, the first identification network and the second identification network are adjusted respectively according to the first loss function value, the second loss function value and the third loss function value.

6. The method according to claim 1, characterized in that The step of determining a parking space based on the clustered corner points includes: Based on the parallelogram geometric features, the geometric positions of the clustered corner points are corrected to determine the parking spaces.

7. A parking space detection device based on multi-corner clustering, characterized in that: The device comprises: An image acquisition module is used to acquire the target image to be detected captured by the vehicle-mounted camera; A map extraction module is configured to extract multiple heat maps and label maps from the target image, wherein each heat map corresponds to a corner point direction, the heat map contains heat values ​​indicating that different pixels belong to the corner point in the corresponding corner point direction, and the label map contains label values ​​indicating that different pixels belong to parking spaces; The parking space detection module is used to extract multiple target pixel points whose heat values ​​meet the corresponding threshold from each heat map, and obtain the label value of each target pixel point based on the label map; cluster the multiple target pixel points whose label values ​​meet the corresponding difference conditions into corner points of the same parking space in different corner point directions, and determine the parking space based on the clustered corner points.

8. The device according to claim 7, characterized in that The graph extraction module is specifically used to: Retrieving a pre-trained detection model, wherein the detection model is obtained by training a neural network with the goal of predicting a heat map of a sample image close to a heat map annotation result of the sample image and predicting a label map of the sample image close to a label map annotation result of the sample image; The target image is input into the detection model to obtain multiple heat maps and label maps output by the detection model for the target image.

9. An electronic device, characterized in that: The electronic device includes: at least one memory and at least one processor; the memory stores a program, the processor calls the program stored in the memory, and the program is used to implement the parking space detection method based on multi-corner point clustering according to any one of claims 1-6.

10. A storage medium, characterized in that: The storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the parking space detection method based on multi-corner point clustering according to any one of claims 1 to 6.

Images