Parking space detection method and device, electronic equipment and storage medium

Abstract

The application provides a parking space detection method and device, electronic equipment and a storage medium. By extracting a heat map and an offset vector map of an image under different feature point types, a corner point can be predicted to detect a parking space. Based on the application, horizontal, vertical, and arbitrary angle inclined parking spaces can be accurately identified, and the application has high reliability, high precision, and intelligent parking space detection performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving and automatic parking, and more particularly to a parking space detection method and device, an electronic device and a storage medium. BACKGROUND

[0002] The parking space detection technology is one of the key technologies of the vehicle automatic parking system, and is a prerequisite for accurately planning a parking path and accurately parking in a parking space.

[0003] In the parking space detection technology, the detection of a linear parking space completely relies on a camera sensor, and a machine vision algorithm is used to detect the specific position of the parking space. Early visual perception algorithms mainly artificially set some graphic features to find related information in a picture and extract a parking space. However, the parking space scene is very complex, especially in a community where there is no unified standard parking space feature, such as various colors of lines, grass, and brick ground, and it is difficult to cover all scenes through artificial design. SUMMARY

[0004] In view of this, to solve the above problems, the present application provides a parking space detection method, device, electronic device and storage medium, and the technical solutions are as follows:

[0005] In one aspect, the present application provides a parking space detection method, which comprises:

[0006] acquiring a target image to be detected collected by a vehicle-mounted camera;

[0007] extracting a plurality of heat maps and offset vector maps of the target image, one heat map corresponding to one feature point type, the feature point type including a corner point and a center point, the heat map containing heat values of different pixel points belonging to the corresponding feature point type, and the offset vector map containing interval distances between different pixel points as center points and corner points;

[0008] extracting a first pixel point with a heat value satisfying a corresponding threshold from a first heat map, and acquiring a target interval distance corresponding to the first pixel point based on the offset vector map, so as to locate a second pixel point as a corner point of a target parking space by using the target interval distance, the feature point type corresponding to the first heat map being a center point, and the target parking space being a parking space with the first pixel point as the center point;

[0009] acquiring a target heat value associated with the second pixel point based on a second heat map, and correcting the corner point of the target parking space by using the target heat value, the feature point type corresponding to the second heat map being a corner point.

[0010] Optionally, the extracting of the plurality of heat maps and offset vector maps of the target image comprises:

[0011] retrieve a detection model pre-trained, the detection model is obtained by training a neural network, and the detection model is trained to make a heat map prediction result of a sample image close to a heat map label result of the sample image and make an offset vector map prediction result of the sample image close to an offset vector map label result of the sample image.

[0012] input the target image into the detection model to obtain a plurality of heat maps and offset vector maps output by the detection model for the target image.

[0013] Optionally, the sample image is obtained in the following manner:

[0014] obtain a first daytime sample image;

[0015] retrieve a conversion model pre-trained based on a generative adversarial network technology, and convert the first daytime sample image into a first nighttime sample image through the conversion model, the conversion model is obtained by training a generative adversarial network, and the conversion model is trained to make a conversion result of a second daytime sample image close to a second nighttime sample image labeled for the second daytime sample image.

[0016] Optionally, the generative adversarial network comprises a first generative network, a first discriminative network corresponding to the first generative network, a second generative network, and a second discriminative network corresponding to the second generative network.

[0017] Correspondingly, the process of training the conversion model based on the generative adversarial network technology comprises:

[0018] input the second daytime sample image into the first generative network and the second discriminative network respectively, and input the second nighttime sample image into the first discriminative network and the second generative network, so as to:

[0019] the first generative network generates a third nighttime sample image based on the second daytime sample image and outputs to the first discriminative network, so that the first discriminative network processes the second nighttime sample image and the third nighttime sample image; the second generative network generates a third daytime sample image based on the second nighttime sample image and outputs to the second discriminative network, so that the second discriminative network processes the third daytime sample image and the second daytime sample image.

[0020] calculate a first loss function value according to the processing result of the first discriminative network, and calculate a second loss function value according to the processing result of the second discriminative network.

[0021] adjust network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network according to the first loss function value and the second loss function value respectively;

[0022] in a case where a corresponding training end condition is met at present, taking the adjusted first generation network as the conversion model.

[0023] Optionally, the method further comprises:

[0024] inputting the third night sample image into the second generation network and inputting the third day sample image into the first generation network to achieve:

[0025] the second generation network generates a fourth day sample image based on the third night sample image; and the first generation network generates the fourth night sample image based on the third day sample image;

[0026] calculating a third loss function value according to a comparison result of the second day sample image and the fourth day sample image and a comparison result of the second night sample image and the fourth night sample image;

[0027] Correspondingly, the adjusting of the network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network according to the first loss function value and the second loss function value respectively comprises:

[0028] adjusting the network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network according to the first loss function value, the second loss function value and the third loss function value respectively.

[0029] Optionally, the obtaining of the target heat value associated with the second pixel point based on the second heat map comprises:

[0030] determining a first heat value corresponding to the second pixel point and a second heat value corresponding to a plurality of third pixel points centered on the second pixel point based on the second heat map;

[0031] selecting a maximum heat value from the first heat value and the second heat value as the target heat value;

[0032] Correspondingly, the correcting of the corner point of the target parking space by using the target heat value comprises:

[0033] taking the pixel point corresponding to the target heat value as the actual corner point of the target parking space.

[0034] Another aspect of the present application provides a parking space detection device, which comprises:

[0035] an image acquisition module, configured to acquire a target image to be detected collected by a vehicle-mounted camera;

[0036] an image extraction module, configured to extract a plurality of heat maps and offset vector maps of the target image, one heat map corresponding to one feature point type, the feature point type including a corner point and a center point, the heat map containing heat values of different pixel points belonging to the corresponding feature point type, and the offset vector map containing interval distances between different pixel points as center points and corner points;

[0037] a parking space detection module, configured to extract a first pixel point with a heat value satisfying a corresponding threshold from a first heat map, and acquire a target interval distance corresponding to the first pixel point based on the offset vector map, so as to locate a second pixel point as a corner point of a target parking space by using the target interval distance, the feature point type corresponding to the first heat map being a center point, and the target parking space being a parking space with the first pixel point as the center point; acquire a target heat value associated with the second pixel point based on a second heat map, and correct the corner point of the target parking space by using the target heat value, the feature point type corresponding to the second heat map being a corner point.

[0038] Optionally, the image extraction module is specifically configured to:

[0039] invoke a detection model obtained by pre-training, the detection model being obtained by training a neural network with a target of making a heat map prediction result of a sample image close to a heat map label result of the sample image and making an offset vector map prediction result of the sample image close to an offset vector map label result of the sample image, and input the target image into the detection model to acquire a plurality of heat maps and offset vector maps output by the detection model for the target image.

[0040] In another aspect, the present application provides an electronic device, comprising at least one memory and at least one processor; the memory stores a program, and the processor invokes the program stored in the memory, and the program is used to implement the parking space detection method.

[0041] In another aspect, the present application provides a storage medium, wherein the storage medium stores computer executable instructions, and the computer executable instructions are used to execute the parking space detection method.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] The application provides a parking space detection method and device, electronic equipment and a storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0045] Figure 1 The method flowchart of the parking space detection method provided by the embodiment of the present application is shown in the figure.

[0046] Figure 2 The corner point labeling schematic diagram provided by the embodiment of the present application is shown in the figure.

[0047] Figure 3 The neural network prediction schematic diagram provided by the embodiment of the present application is shown in the figure.

[0048] Figure 4 The structure schematic diagram of the generative adversarial network provided by the embodiment of the present application is shown in the figure.

[0049] Figure 5 The structure schematic diagram of the parking space detection device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0051] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0052] Parking space detection is one of the key technologies of vehicle automatic parking system, and is the premise of accurate planning of parking path and accurate parking into parking space. The parking space detection technology can usually be realized by using ultrasonic radar, camera, laser radar, GPS and other sensor devices, wherein:

[0053] 1) Ultrasonic radar: the main sensor relied on by automatic parking systems in recent years, but only recognizes parking spaces composed of front and rear (left and right) vehicles, and is powerless for parking spaces with clear parking lines but no front and rear (left and right) vehicles. Although radar can accurately detect distance, the fan-shaped characteristics of radar waves make it impossible to accurately identify the shape of obstacles. Currently, the edge coordinate values of front and rear obstacles are compensated by manually setting compensation values, and the error value of the parking edge is about 15 cm, which has a great impact on parking accuracy, parking path planning, and parking effect

[0054] 2) Laser radar: high cost, and currently cannot be mass-produced;

[0055] 3) Civil GPS: poor signal accuracy, cannot meet the requirements of parking space recognition accuracy, and cannot be used in underground parking lots and other places;

[0056] 4) Camera: the biggest advantage of machine vision is that it can recognize the shape of an object. Currently, parking spaces in regular parking lots are marked with parking lines, and the first three sensors cannot detect parking line markings, only cameras can detect line parking.

[0057] In parking space detection technology, line parking detection completely relies on camera sensors, and machine vision algorithms are used to detect the specific location of the parking space. Early visual perception algorithms mainly manually set some graphical features to find related information in the picture and extract the parking space. However, the parking space scene is very complex, especially in communities where there is no unified standard parking feature, such as various colored lines, grass, brick, etc., making it difficult to cover all scenarios through manual design.

[0058] Current parking space detection schemes based on big data mainly fall into two directions:

[0059] 1) Detect parking lines: but often encounter scenes with severely damaged lines or unclear non-painted lines, so line-based parking detection currently has a very low detection rate in dim light and unclear parking line environments.

[0060] 2) Detect parking corner points: some deep learning vision methods can detect parking corner point features based on a large amount of parking scene data, but for so many parking corner points in a picture, which corner points can form a parking space requires complex logic to associate, and the performance is not very good in multiple scenarios, and the angle of the diagonal parking space (the angle between the entrance line and the parking side line) cannot be reliably detected, generally only supporting standard rectangular parking spaces.

[0061] In view of the problems that the current parking space detection has poor complex scene recognition performance, the visual deep learning method is difficult to associate four corner points of the same parking space, and cannot detect an arbitrary angle inclined parking space, the application provides an end-to-end parking space corner point prediction and association scheme based on deep learning, which can further improve the reliability of parking space detection in complex scenes, and can improve the accuracy of automatic identification of parking space types (horizontal parking space, vertical parking space or inclined parking space at any angle), and no longer requires the customer to select, greatly enhancing the user experience. In addition, the application mainly uses the original automatic parking and 360-degree surround hardware platform of the vehicle, and develops the algorithm from the software level, without additional hardware cost, and has high cost performance.

[0062] Referring to Figure 1 The application provides a parking space detection method, and the method comprises the following steps:

[0063] S10, acquiring a target image to be detected collected by a vehicle-mounted camera.

[0064] In the embodiment of the application, taking the fisheye camera as an example, the fisheye images collected by the four fisheye cameras in front, behind, left and right of the vehicle can be spliced to obtain a bird's eye view centered on the vehicle, which is used as the target image for subsequent detection of the parking space.

[0065] S20, extracting a plurality of heat maps and offset vector maps of the target image, one heat map corresponding to one feature point type, the feature point type including a corner point and a center point, the heat map containing heat values of different pixel points belonging to the corresponding feature point type, and the offset vector map containing interval distances of different pixel points as the center point and the corner point.

[0066] In the embodiment of the application, the deep learning neural network can be used to perform end-to-end detection on the target image to obtain a plurality of heat maps and offset vector maps (hereinafter referred to as offset maps) of the target image. In addition, the number of the extracted plurality of heat maps is at most three, because one parking space has three feature point types, including: the front corner point feature is similar, so two front corner points are classified as the feature point type of the front corner point, and two rear corner points are classified as the feature point type of the rear corner point, and the vehicle center point is classified as the feature point type of the center point.

[0067] It should be noted that all the corner points in the embodiment of the application are not the inner corner points of the parking space but the corner points intersecting between the parking spaces, that is, each parking space does not necessarily have a separate front corner point and a rear corner point, and a front corner point and a rear corner point can be shared between two parking spaces. This is because the idea of the application is to first determine the center point of the parking space, and then determine the corner point of the parking space based on the center point of the parking space.

[0068] In the specific implementation process, step S20, "extracting multiple heatmaps and offset vector maps of the target image," can be performed using the following steps:

[0069] The pre-trained detection model is retrieved. The detection model is trained on a neural network with the goal of predicting the heat map of the sample image to be close to the heat map annotation result of the sample image and predicting the offset vector map of the sample image to be close to the offset vector annotation result of the sample image.

[0070] The target image is input into the detection model to obtain multiple heatmaps and offset vector maps output by the detection model for the target image.

[0071] In this embodiment of the invention, when labeling corner points on sample images, the labels can be placed in either a counter-clockwise or clockwise order, so the labeling order used for the sample images is the same. See also Figure 2 The diagram shows the corner point markings. Regardless of the relative position of the vehicle and the parking space, the corner points of the parking space are marked counterclockwise. During the counterclockwise search, the first corner point encountered is designated as Corner Point 1. Continuing counterclockwise, the second corner point encountered is Corner Point 2, and so on, until Corner Point 3, Corner Point 4, and so on. Corner Points 1 and 2 are marked as "front corner points," while Corner Points 3 and 4 are marked as "rear corner points." Furthermore, the position of the center point of the parking space can be calculated based on the positions of its four corner points.

[0072] Furthermore, the embodiments of the present invention train the neural network in two aspects:

[0073] 1) For the labeled sample images, the above three feature point types are defined as hotspots using a Gaussian distribution. The Gaussian distribution calculation method for the heat value is as follows:

[0074]

[0075] Wherein, HeatValue is the heat value of a pixel. The larger the value, the greater the probability that the pixel belongs to a certain feature point type; σ is a hyperparameter representing the pixel distance factor; x0, y0 are the coordinates of the labeled point (a point of the labeled corner point). This point can be a sub-pixel point, and x, y are points within the range of σ near the labeled point.

[0076] In addition, in this embodiment of the invention, when calculating the heat value, a rectangular region of interest is created by extending 3*σ pixels in each of the four directions (up, down, left, and right) of the marked point (if an image boundary is encountered, it is truncated at the boundary). All points within the rectangular region participate in the calculation of the heat value. The closer a point is to the marked point, the greater the probability that it is a marked point and the larger the calculated heat value. Similarly, the farther a point is from the marked point, the less likely it is to be a marked point and the smaller the calculated heat value.

[0077] The three feature point types for parking spaces—"front corner point," "rear corner point," and "center point"—correspond to three deep learning computation channels. The neural network regresses the corner points labeled in the sample images, using the Mean Square Error (MSE) loss function for model supervision. The MSE loss function expression is:

[0078]

[0079] Where m is the number of pixels involved in calculating MSE, and y i f(x) is the true value. i ) represents the neural network prediction value.

[0080] For example, for a sample image, ground truth heatmaps are generated on a 104*104*3 matrix, where '3' represents three channels: the front corner heatmap (feature point type is front corner), the back corner heatmap (feature point type is back corner), and the center point heatmap (feature point type is center). The front corner, back corner, and center points in the sample image are defined as hotspots using a Gaussian distribution. The neural network regresses these ground truth results, using the Mean Square Error (MSE) loss function for model supervision.

[0081] 2) The heatmap detection technology mentioned above only detects the front corner, rear corner, and center point of a parking space, but cannot determine which points constitute a parking space. However, if the predicted value of the center point is greater than the corresponding threshold, it can be determined that there is a parking space at that location. Then, by predicting the distance between the center point and the four corner points of the parking space (i.e., the offset ground truth value), the four corner points can be correlated.

[0082] Therefore, this invention designs a neural network to regress the offset values ​​from the center point of a parking space to its four corner points. The actual parking environment is affected by many factors such as time, weather, and image clarity, and the prediction results of a heatmap cannot accurately predict the center point of the parking space. Therefore, it is necessary to calculate the distance between corner points and points near the center point within the area where the center point is located.

[0083] The calculation formula is as follows:

[0084]

[0085] Where i represents the parking space corner point code, namely the front right corner point, front left corner point, rear left corner point, and rear right corner point, and x represents the parking space corner point code. i y i Let x be the coordinates of the corner point of parking space i. o y o The offset_x represents the coordinates of the center point of the parking space. i Offset_y is the distance in the x-direction from the corner point to the center point. i This represents the distance in the y-direction from the corner point to the center point.

[0086] The loss function for regressing the corner positions from the central region points is the Smooth L1 function:

[0087]

[0088] For example, for a sample image, the offset truth value is generated on a matrix of size 104*104*8, where '8' represents 8 channels, each channel is 104*104, and the values ​​represent the distances from the four corner points of the parking space to the center point of the parking space in the x and y directions of the image, respectively.

[0089] See Figure 3 The diagram illustrates neural network prediction. An image is input into the neural network, which has dual outputs: one output is a heatmap with three channels, and the other output is an offset image. This embodiment of the invention does not limit the network structure of the neural network.

[0090] It should be noted that in the offset map output by the neural network prediction, pixels with a low probability of being the center point can be omitted from the display. Only the pixels predicted to be the center point and the true offset values ​​between the pixel and the two front and two back corner points when the pixel is the center point are displayed.

[0091] It should also be noted that, for the sample image, which is an RGB image with a size of 416*416*3, in order to improve the robustness of the algorithm, this embodiment of the invention can perform random data enhancement operations on the sample image in six aspects: translation, rotation, hue, saturation, brightness, and contrast. The sample image is then input into the neural network to obtain dual-channel output.

[0092] In addition, deep learning technology has made great progress in the field of visual perception in recent years. By using data-driven algorithm models, neural networks can learn a variety of visual environment features under data training, thereby providing robustness of the algorithm in various scenarios and greatly improving the performance of visual parking space detection.

[0093] Data-driven algorithms require a large amount of data as their foundation, providing diverse scenarios to enhance performance. Therefore, before training a deep learning neural network, traditional methods perform data augmentation, such as image translation, rotation, scaling, modifying contrast, saturation, and brightness, and adding random noise to diversify the training images. However, these augmentation operations still fail to address the impact of different natural lighting conditions on scenes, and cannot realistically simulate changes in scene lighting.

[0094] Parking space detection data collection typically involves dedicated data collection vehicles deployed in various parks, residential areas, underground garages, and other locations with parking spaces. The time of day is the most significant influencing factor; however, most data collection occurs during the day, resulting in a dominant proportion of daytime data in above-ground scenarios. While network model predictions perform well during the day, they deteriorate under nighttime conditions. This is because outdoor lighting conditions differ significantly between day and night, leading to noticeable differences in the images captured by cameras. For deep learning neural networks, a richer and more diverse training dataset is preferable. However, collecting data from the same residential area at different times increases the workload by more than two times, as does the time required for image acquisition, significantly impacting project development timelines.

[0095] Therefore, current parking space detection algorithms based on surround-view stitching often suffer from poor performance in nighttime scenes due to the predominantly daytime ground-level data collection. Relying entirely on data collection increases costs, data annotation, and prolongs project development cycles. To address this, this invention transforms daytime data into nighttime scenes with and without streetlights, considering not only changes in image brightness but also the overall impact of vehicle headlights and streetlights on the image. This generates nighttime images that are imperceptible to the human eye. By expanding the dataset based on daytime data, it reduces data collection costs, shortens development cycles, and improves development efficiency.

[0096] The following describes the method for acquiring sample images in embodiments of the present invention, including the following steps:

[0097] Acquire the first daytime sample images;

[0098] The transformation model, which was pre-trained based on generative adversarial network technology, is retrieved to transform the first daytime sample image into the first nighttime sample image. The transformation model is trained on the generative adversarial network with the goal of making the transformation result of the second daytime sample image close to the second nighttime sample image labeled with the second daytime sample image.

[0099] In this embodiment of the invention, a generative adversarial network is trained to generate a conversion model using second daytime sample images and second nighttime sample images. For scenarios requiring no streetlights at night, the second nighttime sample image is a nighttime sample image with no streetlights; conversely, for scenarios requiring streetlights at night, the second nighttime sample image is a nighttime sample image with streetlights. The number of second nighttime sample images for both scenarios is greater than 1000, and the number of second nighttime sample images is consistent across both scenarios. Of course, a conversion model trained on two scenarios can convert the same daytime sample image to obtain nighttime sample images for both scenarios. This embodiment of the invention illustrates the training process of the conversion model using one scenario; the same applies to another scenario.

[0100] See Figure 4 The diagram shows the structure of a Generative Adversarial Network (GAN). A GAN comprises two network structures: a generator network and a discriminator network. The generator network generates an image in the target style from the input image, while the discriminator network determines whether the image output by the generator network is similar to the target style.

[0101] See also Figure 4 In this embodiment of the invention, the generative adversarial network includes a first generative network G. A , and the first generator network G A The corresponding first discrimination network D B Second Generator Network G B And the second discrimination network D corresponding to the second generator network. A Accordingly, the process of training a transformation model based on generative adversarial network technology includes:

[0102] The second daytime sample images are input into the first generation network and the second discrimination network, respectively, and the second nighttime sample images are input into the first discrimination network and the second generation network, to achieve the following:

[0103] The first generation network generates a third nighttime sample image based on the second daytime sample image and outputs it to the first discrimination network so that the first discrimination network can process the second nighttime sample image and the third nighttime sample image; the second generation network generates a third daytime sample image based on the second nighttime sample image and outputs it to the second discrimination network so that the second discrimination network can process the third daytime sample image and the second daytime sample image.

[0104] The first loss function value is calculated based on the processing result of the first discrimination network, and the second loss function value is calculated based on the processing result of the second discrimination network;

[0105] The network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network are adjusted according to the first loss function value and the second loss function value, respectively.

[0106] If the corresponding training termination conditions are met, the adjusted first generator network will be used as the conversion model.

[0107] Suppose that the second daytime sample image is X, and the second nighttime sample image is a ground scene without streetlights at night, denoted as Y. To generate nighttime data using daytime parking data, X is input into the first generator network G. A In the first generator network G A The output is G A (X), it is necessary to determine the first generator network G A The output image G A Whether (X) meets the scenario requirements needs to be input into the first discrimination network D. B In the first discrimination network D B Process the input Y and G A (X) The identification process is complete. The identification metric of the first identification network DB is the following adversarial loss function:

[0108]

[0109] Additionally, Y is input into the second generator network G. B In the middle, the second generator network G B The output is G B (Y), it is necessary to determine the second generator network G. B The output image G B Whether (Y) meets the scenario requirements needs to be input into the second discrimination network D. A In the middle, the second discrimination network D A Process the input X and G B (Y) completes identification, second identification network D A The identification metric is the following adversarial loss function:

[0110]

[0111] Therefore, the first loss function value can be calculated according to the above formula (5), and the second loss function value can be calculated according to the above formula (6). In this embodiment of the invention, the sum of the first loss function value and the second loss function value is used as the loss function value of the generative adversarial network, and then the gradient descent algorithm is used to converge the loss function value of the generative adversarial network to adjust the network parameters of each network structure in the generative adversarial network.

[0112] In addition, the training termination condition can be set based on the training duration or the number of training sessions, and this embodiment of the invention does not limit this.

[0113] Furthermore, to improve the performance of the conversion model, the embodiments of the present invention further consider:

[0114] If daytime data can generate nighttime data, and nighttime data can also generate daytime data, then theoretically:

[0115] G B (G A (x))≈x (7)

[0116] G A (G B (y))≈y (8)

[0117] To address this, the present invention incorporates a recurrent loss function to assist in neural network training:

[0118]

[0119] Therefore, in this embodiment of the invention, the total loss of the generative adversarial network is:

[0120] LOSS = Loss GAN (G A D A (X, Y) + Loss GAN (G B D B (X, Y) + γ*Loss cyc (G A G B (10)

[0121] Where γ is a preset weighting coefficient.

[0122] Based on this, embodiments of the present invention further include the following steps:

[0123] The third nighttime sample image is input into the second generation network, and the third daytime sample image is input into the first generation network to achieve the following:

[0124] The second generation network generates the fourth daytime sample image based on the third nighttime sample image; the first generation network generates the fourth nighttime sample image based on the third daytime sample image.

[0125] The third loss function value is calculated based on the comparison results between the second day sample image and the fourth day sample image, as well as the comparison results between the second night sample image and the fourth night sample image.

[0126] Accordingly, the network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network are adjusted based on the first loss function value and the second loss function value, respectively, including:

[0127] The network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network are adjusted according to the first loss function value, the second loss function value, and the third loss function value, respectively.

[0128] In this embodiment of the invention, the third loss function value is calculated according to the above formula (9), and then the sum of the first loss function value, the second loss function value and the third loss function value is used as the loss function value of the generative adversarial network according to the above formula (10). Then, the gradient descent algorithm is used to converge the loss function value of the generative adversarial network to adjust the network parameters of each network structure in the generative adversarial network.

[0129] S30, extract the first pixel point whose heat value meets the corresponding threshold from the first heat map, and obtain the target interval distance corresponding to the first pixel point based on the offset vector map, so as to locate the second pixel point as the corner point of the target parking space using the target interval distance. The feature point type corresponding to the first heat map is the center point, and the target parking space is the parking space with the first pixel point as the center point.

[0130] In this embodiment of the invention, see also Figure 3 The diagram illustrates neural network prediction. For the three heatmaps output by the detection model, the first pixel with a heatmap feature point type of center point (i.e., the first heatmap) is selected, and its heat value is greater than the corresponding threshold. For example, the heat values ​​of different pixels in the first heatmap can be sorted from smallest to largest, and the pixels corresponding to the K largest heat values ​​are taken as the first pixels. Correspondingly, a first pixel is positioned as a center point. Based on this, for a parking space within the vehicle's environment, the parking space can be located based on its center point. Here, K represents the maximum number of parking spaces to detect.

[0131] Furthermore, for each first pixel, the offset true values ​​of the corresponding four corner points are obtained from the offset image. Based on the offset true values, the four corner points (i.e., the second pixel) of the parking space with the first pixel as the center point can be located, thus completing the coarse positioning of the parking space corner points.

[0132] S40: Obtain the target heat value associated with the second pixel based on the second heat map, and use the target heat value to correct the corner point of the target parking space. The feature point type corresponding to the second heat map is corner point.

[0133] In this embodiment of the invention, the corner points that are coarsely located can be divided into front corner points and back corner points. Therefore, for the second pixel point, its corresponding heat value can be obtained from the heat map of the front corner point / back corner point. This heat value can be used as the target heat value. If the target heat value is greater than the corresponding threshold, it can be considered to have high confidence and be used as the front corner point / back corner point. Otherwise, if it has low confidence, it can be removed. Then, the parallelogram geometric features are used to perform geometric positioning of the parking space.

[0134] In the specific implementation process, when determining the target heat value associated with the second pixel, the point with the highest confidence near the coarsely located corner point can be selected as the fine-tuning corner point position for the parking space. Specifically, on the one hand, the first heat value corresponding to the second pixel in the front / back corner heat map can be determined; on the other hand, a region centered on the second pixel can be determined, such as the other pixels in a 5*5 region besides the second pixel, i.e., the second heat value corresponding to the third pixel. Then, the maximum value among the first heat value and multiple second heat values ​​is selected as the target heat value to complete the fine-tuning of the parking space corner point position, thereby matching the fine-tuned corner points and locating the parking space.

[0135] In summary, this invention, through an end-to-end neural network, can accurately associate corner points of the same parking space while detecting them, and can also identify angled parking spaces at any angle. This greatly improves the adaptability of the automatic parking system to different parking space scenarios and enhances the user experience.

[0136] Based on the parking space detection method provided in the above embodiments, the present invention provides a corresponding apparatus for performing the above parking space detection method, the structural schematic diagram of which is shown below. Figure 5 As shown, it includes:

[0137] Image acquisition module 10 is used to acquire the target image to be detected captured by the vehicle-mounted camera;

[0138] Image extraction module 20 is used to extract multiple heatmaps and offset vector maps of the target image. Each heatmap corresponds to a feature point type, which includes corner points and center points. The heatmap contains heat values ​​of different pixels belonging to the corresponding feature point type, and the offset vector map contains the distance between different pixels when they are used as center points and corner points.

[0139] The parking space detection module 30 is used to extract a first pixel point whose heat value meets the corresponding threshold from the first heat map, and obtain the target interval distance corresponding to the first pixel point based on the offset vector map, so as to locate the second pixel point as the corner point of the target parking space using the target interval distance. The feature point type corresponding to the first heat map is the center point, and the target parking space is the parking space with the first pixel point as the center point; the module obtains the target heat value associated with the second pixel point based on the second heat map, and corrects the corner point of the target parking space using the target heat value. The feature point type corresponding to the second heat map is the corner point.

[0140] Optional, the image extraction module 20 is specifically used for:

[0141] The pre-trained detection model is retrieved. The detection model is trained on a neural network with the goal of predicting that the heatmap prediction result of the sample image is close to the heatmap annotation result of the sample image and predicting that the offset vector prediction result of the sample image is close to the offset vector annotation result of the sample image. The target image is input into the detection model to obtain multiple heatmaps and offset vector maps output by the detection model for the target image.

[0142] Optionally, the image extraction module 20 may acquire sample images in the following ways:

[0143] Acquire the first daytime sample image; retrieve the transformation model pre-trained based on generative adversarial network technology, and use the transformation model to transform the first daytime sample image into the first nighttime sample image. The transformation model is trained on the generative adversarial network with the goal of making the transformation result of the second daytime sample image close to the second nighttime sample image labeled with the second daytime sample image.

[0144] Optionally, the generative adversarial network includes a first generative network, a first authentication network corresponding to the first generative network, a second generative network, and a second authentication network corresponding to the second generative network;

[0145] Correspondingly, the process of training the transformation model based on generative adversarial network technology in the graph extraction module 20 includes:

[0146] The second daytime sample image is input into the first generator network and the second discriminator network, and the second nighttime sample image is input into the first discriminator network and the second generator network, respectively, to achieve the following: the first generator network generates a third nighttime sample image based on the second daytime sample image and outputs it to the first discriminator network, so that the first discriminator network processes the second nighttime sample image and the third nighttime sample image; the second generator network generates a third daytime sample image based on the second nighttime sample image and outputs it to the second discriminator network, so that the second discriminator network processes the third daytime sample image and the second daytime sample image; a first loss function value is calculated based on the processing result of the first discriminator network, and a second loss function value is calculated based on the processing result of the second discriminator network; the network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network are adjusted according to the first loss function value and the second loss function value, respectively; and when the corresponding training termination condition is met, the adjusted first generator network is used as the conversion model.

[0147] Optionally, the graph extraction module 20 is also used for:

[0148] The third nighttime sample image is input into the second generation network, and the third daytime sample image is input into the first generation network, so as to achieve: the second generation network generates a fourth daytime sample image based on the third nighttime sample image; the first generation network generates a fourth nighttime sample image based on the third daytime sample image; and the third loss function value is calculated based on the comparison results of the second daytime sample image and the fourth daytime sample image, and the comparison results of the second nighttime sample image and the fourth nighttime sample image.

[0149] Accordingly, the process by which the graph extraction module 20 adjusts the network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network based on the first loss function value and the second loss function value includes:

[0150] The network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network are adjusted according to the first loss function value, the second loss function value, and the third loss function value, respectively.

[0151] Optionally, the process by which the parking space detection module 30 obtains the target heat value associated with the second pixel based on the second heat map includes:

[0152] Based on the second heat map, determine the first heat value corresponding to the second pixel and the second heat values ​​corresponding to multiple third pixels centered on the second pixel; select the largest heat value from the first heat value and the second heat value as the target heat value;

[0153] Correspondingly, the process by which the parking space detection module 30 corrects the corner points of the target parking space using the target heat value includes:

[0154] The pixel corresponding to the target heat value is taken as the actual corner point of the target parking space.

[0155] Based on the parking space detection method provided in the above embodiments, this invention also provides an electronic device, which includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, the program being used to implement the parking space detection method.

[0156] Based on the parking space detection method provided in the above embodiments, this embodiment of the invention also provides a storage medium storing computer-executable instructions for executing the parking space detection method.

[0157] The above provides a detailed description of the parking space detection method, device, electronic equipment, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0158] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0159] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that elements inherent to a process, method, article, or apparatus that comprises a list of elements, or elements inherent to such processes, methods, articles, or apparatus, are also included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0160] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A parking space detection method, characterized in that, The method includes: Acquire the target image to be detected captured by the vehicle-mounted camera; Multiple heatmaps and offset vector maps are extracted from the target image. Each heatmap corresponds to a feature point type, which includes corner points and center points. The heatmap contains heat values ​​of different pixels belonging to the corresponding feature point type, and the offset vector map contains the distance between different pixels when they are used as center points and corner points. First pixel points whose heat values ​​meet the corresponding thresholds are extracted from the first heat map, and the target interval distance corresponding to the first pixel points is obtained based on the offset vector map. The second pixel points, which are the corner points of the target parking spaces, are located using the target interval distance. The feature point type corresponding to the first heat map is the center point, and the target parking space is the parking space with the first pixel point as the center point. The target heat value associated with the second pixel is obtained based on the second heat map, and the corner point of the target parking space is corrected using the target heat value. The feature point type corresponding to the second heat map is a corner point.

2. The method according to claim 1, characterized in that, The extraction of multiple heatmaps and offset vector maps of the target image includes: The detection model obtained through pre-training is retrieved. The detection model is trained on a neural network with the goal of predicting the heat map of the sample image to be close to the heat map annotation result of the sample image and predicting the offset vector map of the sample image to be close to the offset vector annotation result of the sample image. The target image is input into the detection model to obtain multiple heatmaps and offset vector maps output by the detection model for the target image.

3. The method according to claim 2, characterized in that, The methods for acquiring the sample images include: Acquire the first daytime sample images; A conversion model pre-trained based on generative adversarial network technology is retrieved to convert the first daytime sample image into a first nighttime sample image. The conversion model is trained on a generative adversarial network with the goal of converting the second daytime sample image to a second nighttime sample image labeled with the second daytime sample image.

4. The method according to claim 3, characterized in that, The generative adversarial network includes a first generative network, a first authentication network corresponding to the first generative network, a second generative network, and a second authentication network corresponding to the second generative network; Accordingly, the process of training the transformation model based on generative adversarial network technology includes: The second daytime sample image is input into the first generation network and the second discrimination network, respectively, and the second nighttime sample image is input into the first discrimination network and the second generation network, to achieve the following: The first generation network generates a third nighttime sample image based on the second daytime sample image and outputs it to the first discrimination network so that the first discrimination network processes the second nighttime sample image and the third nighttime sample image; the second generation network generates a third daytime sample image based on the second nighttime sample image and outputs it to the second discrimination network so that the second discrimination network processes the third daytime sample image and the second daytime sample image. The first loss function value is calculated based on the processing result of the first discrimination network, and the second loss function value is calculated based on the processing result of the second discrimination network; Adjust the network parameters of the first generator network, the second generator network, the first discrimination network, and the second discrimination network according to the first loss function value and the second loss function value, respectively. If the corresponding training termination condition is met, the adjusted first generator network will be used as the conversion model.

5. The method according to claim 4, characterized in that, The method further includes: The third nighttime sample image is input into the second generation network, and the third daytime sample image is input into the first generation network to achieve the following: The second generation network generates a fourth daytime sample image based on the third nighttime sample image; the first generation network generates a fourth nighttime sample image based on the third daytime sample image. The third loss function value is calculated based on the comparison results between the second daytime sample image and the fourth daytime sample image, and the comparison results between the second nighttime sample image and the fourth nighttime sample image. Accordingly, adjusting the network parameters of the first generator network, the second generator network, the first discriminator network, and the second discriminator network based on the first loss function value and the second loss function value respectively includes: The network parameters of the first generator network, the second generator network, the first discrimination network, and the second discrimination network are adjusted according to the first loss function value, the second loss function value, and the third loss function value, respectively.

6. The method according to claim 1, characterized in that, The step of obtaining the target heat value associated with the second pixel based on the second heat map includes: Based on the second heat map, determine the first heat value corresponding to the second pixel and the second heat value corresponding to a plurality of third pixels centered on the second pixel; Select the largest heat value from the first heat value and the second heat value as the target heat value; Accordingly, the step of correcting the corner point of the target parking space using the target heat value includes: The pixel corresponding to the target heat value is taken as the actual corner point of the target parking space.

7. A parking space detection device, characterized in that, The device includes: The image acquisition module is used to acquire the target image to be detected captured by the vehicle-mounted camera; The image extraction module is used to extract multiple heatmaps and offset vector maps of the target image. Each heatmap corresponds to a feature point type, and the feature point types include corner points and center points. The heatmap contains heat values ​​of different pixels belonging to the corresponding feature point types, and the offset vector map contains the distance between different pixels when they are used as center points and corner points. The parking space detection module is used to extract a first pixel point whose heat value meets the corresponding threshold from a first heat map, and obtain the target interval distance corresponding to the first pixel point based on the offset vector map, so as to locate a second pixel point as the corner point of the target parking space using the target interval distance. The feature point type corresponding to the first heat map is the center point, and the target parking space is the parking space with the first pixel point as the center point. The module also obtains the target heat value associated with the second pixel point based on a second heat map, and uses the target heat value to correct the corner point of the target parking space. The feature point type corresponding to the second heat map is the corner point.

8. The apparatus according to claim 7, characterized in that, The image extraction module is specifically used for: The detection model obtained through pre-training is retrieved. The detection model is trained on a neural network with the goal of predicting the heat map of the sample image to be close to the heat map annotation result of the sample image and predicting the offset vector map of the sample image to be close to the offset vector annotation result of the sample image. The target image is input into the detection model to obtain multiple heatmaps and offset vector maps output by the detection model for the target image.

9. An electronic device, characterized in that, The electronic device includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, the program being used to implement the parking space detection method according to any one of claims 1-6.

10. A storage medium, characterized in that, The storage medium stores computer-executable instructions for executing the parking space detection method according to any one of claims 1-6.

Images