Parking space detectable method, device, electronic equipment and storage medium

Abstract

The application provides a parking space detection method and device, electronic equipment and storage medium. The heat value of each pixel point of an image belonging to the center point of an empty parking space or an occupied parking space is determined by extracting an empty parking space heat map and an occupied parking space heat map of the image, and then target pixel points with heat values meeting corresponding threshold values are extracted from the empty parking space heat map and the occupied parking space heat map, respectively. Whether the parking space with the target pixel point as the center point is occupied is determined according to the heat value of the target pixel point. The application can quickly identify whether a parking space is available, and has the basis for engineering production.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field, more particularly, to a drivable parking space detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] Parking space detection technology is one of the key technologies of vehicle automatic parking system, and is the premise for accurately planning parking path and accurately parking into parking space.

[0003] Since general camera-based algorithms cannot provide information about whether a parking space is occupied, but to provide empty parking space information to users, the usual practice is to use the ranging information of lateral ultrasonic radar to determine whether there is an obstacle in the parking space. If the visual parking area detects an obstacle, it is an occupied parking space, and if it does not detect an obstacle (including the obstacle being at the low end of the parking space, and the ranging distance being insufficient), it is an empty parking space.

[0004] However, in actual application, the waveform of ultrasonic radar is a fan-shaped wave, and is not as precise as laser radar, especially in the vertical parking space scenario. If the parking space is narrow, the ultrasonic echo of the adjacent vehicle in the empty parking space area is often detected, resulting in misjudgment of the empty parking space as an occupied parking space, and poor user experience. SUMMARY

[0005] Therefore, in order to solve the above problems, the present application provides a drivable parking space detection method, device, electronic equipment and storage medium, and the technical solution is as follows:

[0006] In one aspect, the present application provides a drivable parking space detection method, which comprises:

[0007] obtaining a target image to be detected collected by a vehicle-mounted camera;

[0008] extracting an empty parking space heat map and an occupied parking space heat map of the target image, the empty parking space heat map containing heat values of different pixel points belonging to the center point of the empty parking space, and the occupied parking space heat map containing heat values of different pixel points belonging to the center point of the occupied parking space;

[0009] extracting target pixel points with heat values meeting corresponding threshold values from the empty parking space heat map and the occupied parking space heat map, respectively, and determining whether the parking space centered on the target pixel points is occupied according to the heat values of the target pixel points.

[0010] Preferably, the extraction of the empty parking space heat map and the occupied parking space heat map of the target image comprises:

[0011] retrieve a detection model pre-trained, the detection model is trained by a neural network aiming at making a prediction result of an empty parking space heat map of a sample image close to a label result of the empty parking space heat map of the sample image, and making a prediction result of an occupied parking space heat map of the sample image close to a label result of the occupied parking space heat map of the sample image;

[0012] input the target image into the detection model to obtain an empty parking space heat map and an occupied parking space heat map output by the detection model for the target image.

[0013] Preferably, the sample image is obtained in the following manner:

[0014] obtain a first daytime sample image;

[0015] retrieve a conversion model pre-trained based on a generative adversarial network technology, to convert the first daytime sample image into a first nighttime sample image through the conversion model, the conversion model is trained by the generative adversarial network aiming at making a conversion result of a second daytime sample image close to a second nighttime sample image labeled for the second daytime sample image.

[0016] Preferably, the generative adversarial network comprises a first generation network, a first discrimination network corresponding to the first generation network, a second generation network, and a second discrimination network corresponding to the second generation network.

[0017] Correspondingly, the process of training the conversion model based on the generative adversarial network technology comprises:

[0018] input the second daytime sample image into the first generation network and the second discrimination network respectively, and input the second nighttime sample image into the first discrimination network and the second generation network, to realize:

[0019] the first generation network generates a third nighttime sample image based on the second daytime sample image and outputs to the first discrimination network, so that the first discrimination network processes the second nighttime sample image and the third nighttime sample image; the second generation network generates a third daytime sample image based on the second nighttime sample image and outputs to the second discrimination network, so that the second discrimination network processes the third daytime sample image and the second daytime sample image;

[0020] calculate a first loss function value according to a processing result of the first discrimination network, and calculate a second loss function value according to a processing result of the second discrimination network;

[0021] adjust network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network according to the first loss function value and the second loss function value respectively;

[0022] in case that the corresponding training end condition is met, taking the adjusted first generation network as the conversion model.

[0023] Preferably, the method further comprises:

[0024] inputting the third night sample image into the second generation network and inputting the third day sample image into the first generation network to achieve:

[0025] the second generation network generates a fourth day sample image based on the third night sample image; and the first generation network generates the fourth night sample image based on the third day sample image;

[0026] calculating a third loss function value according to a comparison result of the second day sample image and the fourth day sample image and a comparison result of the second night sample image and the fourth night sample image;

[0027] Correspondingly, the adjusting of the network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network according to the first loss function value and the second loss function value respectively comprises:

[0028] adjusting the network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network according to the first loss function value, the second loss function value and the third loss function value respectively.

[0029] Preferably, the determining of whether the parking space centered on the target pixel point is occupied according to the heat value of the target pixel point comprises:

[0030] if the target pixel point is located in both the empty parking space heat map and the occupied parking space heat map, obtaining a first heat value of the target pixel point in the empty parking space heat map and a second heat value of the target pixel point in the occupied parking space heat map respectively;

[0031] selecting the maximum heat value from the first heat value and the second heat value as the heat value of the target pixel point.

[0032] Another aspect of the present application provides a parking space detection device, which comprises:

[0033] an image acquisition module, configured to acquire a target image to be detected collected by a vehicle-mounted camera;

[0034] a heat map extraction module configured to extract an empty parking space heat map and an occupied parking space heat map of the target image, the empty parking space heat map containing heat values of different pixel points belonging to center points of empty parking spaces, and the occupied parking space heat map containing heat values of different pixel points belonging to center points of occupied parking spaces;

[0035] a parking space detection module configured to extract target pixel points with heat values satisfying corresponding threshold values from the empty parking space heat map and the occupied parking space heat map, respectively, and determine whether a parking space with the target pixel point as a center point is occupied according to a heat value of the target pixel point.

[0036] Preferably, the heat map extraction module is specifically configured to:

[0037] retrieve a detection model obtained through pre-training, the detection model being obtained through training of a neural network with a goal that a prediction result of an empty parking space heat map of a sample image approaches a label result of the empty parking space heat map of the sample image and a prediction result of an occupied parking space heat map of the sample image approaches a label result of the occupied parking space heat map of the sample image, and input the target image into the detection model to obtain an empty parking space heat map and an occupied parking space heat map output by the detection model for the target image.

[0038] In another aspect, the present application provides an electronic device, which comprises at least one memory and at least one processor, the memory stores a program, and the processor invokes the program stored in the memory, and the program is used to implement any one of the parking space detection methods.

[0039] In another aspect, the present application provides a storage medium, which stores computer executable instructions, and the computer executable instructions are used to execute any one of the parking space detection methods.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] The present application provides a parking space detection method, device, electronic device and storage medium, which extracts an empty parking space heat map and an occupied parking space heat map of an image, determines heat values of pixel points of the image belonging to center points of empty parking spaces / occupied parking spaces, extracts target pixel points with heat values satisfying corresponding threshold values from the empty parking space heat map and the occupied parking space heat map, respectively, and determines whether a parking space with the target pixel point as a center point is occupied according to a heat value of the target pixel point. The present application can quickly identify whether a parking space is available, and has the basis for engineering production. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description only represent some embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on the provided drawings also belong to the protection scope of the present application.

[0043] Figure 1 The method flowchart of the available parking space detection method provided for the embodiments of the present application is shown in the figure.

[0044] Figure 2 The corner point marking schematic diagram provided for the embodiments of the present application is shown in the figure.

[0045] Figure 3 The neural network prediction schematic diagram provided for the embodiments of the present application is shown in the figure.

[0046] Figure 4 The structure schematic diagram of the generative adversarial network provided for the embodiments of the present application is shown in the figure.

[0047] Figure 5 The structure schematic diagram of the available parking space detection device provided for the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort belong to the protection scope of the present application.

[0049] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Due to the increasing sales of automobiles and the increasing number of automobiles in recent years, the problems of difficult parking and difficult parking have emerged, which promotes the demand for autonomous parking and automatic parking technology, and also promotes the rapid development of this technical field. Parking detection technology is one of the key technologies of vehicle automatic parking system, and is the prerequisite for accurate planning of parking path and accurate parking into parking space. The parking detection technology in recent years is divided into the following two technical solutions:

[0051] 1. Through ultrasonic radar sensor: through radar sensor ranging, combined with vehicle kinematics information, it can be detected whether there is a space on the side of the vehicle to meet the parking demand. The best sensor for this function is laser radar, which has the characteristics of high precision, but due to the high price, it does not meet the cost requirements of mass production, and the current mass production function is to use ultrasonic radar for environment perception;

[0052] 2. Through camera sensor: the above ultrasonic refers to the space parking space, and the ultrasonic radar cannot perceive the relevant information for the parking space with obvious visual line features. After the camera detects, the machine vision algorithm is applied to detect the specific position of the parking space. The early visual perception algorithm mainly manually sets some graphic features to find related information in the picture and extract the parking space. However, the parking space scene is very responsible, especially in the community, there is no unified standard parking feature, such as various colors of lines, grass, brick ground, etc., which is difficult to cover all scenes through manual design. In recent years, deep learning technology has developed greatly in the field of visual perception. Through the data-driven algorithm model, the neural network can learn various visual environmental features under supervised learning to provide the robustness of the algorithm in various scenes.

[0053] As can be seen from the above, ultrasonic radar provides space parking, camera provides line parking, and plays a complementary role. Moreover, the general camera-based algorithm cannot provide information on whether the parking space is occupied, but to provide empty parking space information to the user, the usual way is to use the ranging information of the lateral ultrasonic radar to judge whether there is an obstacle in the parking space. If the visual parking area detects an obstacle, it is an occupied parking space, and if it is not detected (including the obstacle being at the low end of the parking space, the ranging distance being insufficient), it is an empty parking space. However, this algorithm has exposed many problems in actual application, which is due to the fan-shaped wave of ultrasonic radar, which is not as precise as laser radar. Especially in the vertical parking scene, if the parking space is narrow, the ultrasonic echo of the adjacent vehicle in the empty parking area is often detected, and the empty parking space is mistakenly judged as an occupied parking space, which has a poor user experience.

[0054] Humans are easy to judge whether a parking space can park a vehicle, because people have a very strong visual perception system, so the visual perception ability of the camera can be improved to judge whether the parking space is occupied. At present, the industry will adopt two-stage network to judge whether the parking space is occupied:

[0055] First stage: through the extraction of parking angle points and line features, combined with visual post-processing algorithm to extract parking space information, but cannot judge whether the parking space is available. The calculation amount of this neural network is generally small;

[0056] Second stage: using target detection and drivable area detection algorithm to judge whether there is an obstacle in the parking space, but there are three problems in doing so:

[0057] 1. The calculation amount of the target detection and drivable area detection algorithm will be much larger than that of the parking space detection algorithm. The computing power of the parking controller is generally small, and large computing power will reduce the picture calculation frame rate. Under such conditions, in order to meet the function of whether the parking space can be parked, the calculation amount of the parking space detection model must be reduced to meet the calculation frame rate requirement, which will lead to the performance reduction of the parking space detection module;

[0058] 2. The target detection algorithm in the image describes the obstacle through a rectangular Bounding Box and calculates based on the image coordinate system. It detects specific targets such as cars, pedestrians, and cone cylinders. Due to the height of the target, it is impossible to obtain 3D physical distance in a small-power controller. Although the ground point can be calculated by camera calibration parameters, the camera used in the parking space detection is a fisheye camera with large FOV, which has large distortion. The Bounding Box will frame a lot of background, so the positioning of the lower edge is not accurate, and the positioning error will be large.

[0059] 3. The drivable area detection algorithm based on semantic segmentation can perform pixel-level segmentation on the target and calculate the physical distance. However, this algorithm has low reliability in intelligent driving and often has pixel classification segmentation errors. It also requires high data labeling cost and computing resource consumption. Moreover, for low-speed parking spaces, it can only detect the presence of a ground lock and cannot determine the open or closed state, which will affect the system's parking decision.

[0060] 4. Developing multiple algorithms will consume a lot of development time;

[0061] In summary, the algorithm for judging whether the parking space can be parked needs to have basic conditions for engineering mass production, so it needs to have the following characteristics:

[0062] 1. Small computing power, which will not affect the parking space detection algorithm due to the addition of this function;

[0063] 2. Can determine whether there is a vehicle or other obstacle in the parking space;

[0064] 3. Can check the open or closed state of the ground lock.

[0065] The present application aims to solve the pain point that two stages are needed to determine whether a parking space can be parked. Using two neural networks increases the system calculation, reduces the parking space detection frame rate, weakens the parking space detection performance, and the physical distance judgment of the target obstacle is not particularly accurate, and cannot determine whether the ground lock is open or closed. The present application provides a single-stage parking space determination scheme that can be combined with most parking space detection algorithms, and has the above three characteristics and can be engineered for mass production.

[0066] Referring to Figure 1 The embodiment of the present application provides a parking space detection method, which comprises the following steps:

[0067] S10, acquiring a target image to be detected collected by a vehicle-mounted camera.

[0068] In the embodiment of the present application, taking the fisheye camera as an example, the fisheye images collected by the four fisheye cameras in front, back, left and right of the vehicle can be spliced to obtain an aerial view centered on the vehicle as the target image for subsequent detection of the parking space.

[0069] S20, extracting an empty parking space heat map and an occupied parking space heat map of the target image, the empty parking space heat map containing heat values of different pixel points belonging to the center points of the empty parking space, and the occupied parking space heat map containing heat values of different pixel points belonging to the center points of the occupied parking space.

[0070] In the embodiment of the present application, the deep learning neural network can be used to detect whether the parking space is occupied end-to-end to obtain the empty parking space heat map and the occupied parking space heat map of the target image. It should be noted that the occupied parking space not only includes the parking space occupied by the target obstacle such as a vehicle, but also includes the parking space that cannot be parked due to the ground lock being in the open state, i.e. the unparkable parking space is the occupied parking space.

[0071] In the specific implementation process, the step S20 of "extracting the empty parking space heat map and the occupied parking space heat map of the target image" can adopt the following steps:

[0072] The detection model is obtained by training the neural network, and the detection model is trained to approach the sample image empty parking space heat map annotation result by predicting the sample image empty parking space heat map, and to approach the sample image occupied parking space heat map annotation result by predicting the sample image occupied parking space heat map.

[0073] The target image is input into the detection model to obtain the empty parking space heat map and the occupied parking space heat map output by the detection model for the target image.

[0074] In the bird's-eye view stitching panorama, since objects with a height will be distorted away from the camera direction, objects with a height in the parking space (i.e. obstacles affecting parking) will produce obvious deformation (typical scenarios are that there is a car in the parking space, and the ground lock is in the open state), and the occupied parking space area will have obvious feature differences with the empty parking space area in the picture.

[0075] Therefore, by judging the visual feature differences in the bird's-eye view of the empty parking space and the occupied parking space, the application can realize the judgment of whether the parking space can be parked, which solves the problems of the intelligent detection of specific targets by the target detection algorithm and the difficulty in estimating the physical distance, and also solves the problems of low reliability of the drivable area detection by the semantic segmentation and the inability to judge the open and closed state of the ground lock.

[0076] The judgment of feature differences generally adopts the following solutions:

[0077] The parking space position is cut off in the XY axis direction of the picture and input into a small neural network. However, on the one hand, this cutting method will cut some adjacent areas together, affecting the algorithm judgment, and on the other hand, it will cause a two-stage model algorithm (i.e. using one model to identify the parking space, and then using another model to classify the identified parking space to determine whether it is an empty parking space). Therefore, from the perspective of embedded transplantation difficulty and algorithm complexity, it is not a good solution.

[0078] Therefore, the application adopts the following solutions:

[0079] The center point of the parking space is detected, and a secondary attribute label "empty parking space" and "occupied parking space" is assigned, and the judgment of the empty parking space and the occupied parking space is performed by judging the feature of the center area of the parking space.

[0080] For this, in the embodiment of the application, for a sample image, first, the corners are labeled, and the four corners of the parking space in the sample image are labeled in a counterclockwise or clockwise order, and the labeling order of all sample images is the same. Referring to the corner labeling diagram shown in Figure 2 Regardless of the relative position of the vehicle and the parking space, the corners of the parking space are labeled counterclockwise, and in the counterclockwise finding process, the first encountered entry point corner is the first corner 1, the second encountered corner is the second corner 2, and then the third corner 3 and the fourth corner 4 are sequentially labeled. Further, based on the positions of the four corners of the parking space, the position of the center point of the parking space can be further calculated.

[0081] In addition, the sample image is further labeled to determine whether the parking space is occupied. Only the parking space in which the vehicle can be parked without obstacles is an empty parking space. Therefore, each parking space in the sample image has a corresponding center point, and the center point is divided into two types of feature points: the center point of the empty parking space and the center point of the occupied parking space.

[0082] For the two types of feature points, the application defines hot spots through Gaussian distribution, and the HeatValue Gaussian distribution calculation method is as follows:

[0083]

[0084] wherein HeatValue is the heat value of a pixel point, the larger the value, the greater the possibility that the pixel point belongs to the center point of the empty parking space / the center point of the occupied parking space; sigma is a hyperparameter representing a pixel distance factor; x0, y0 are the coordinates of the labeled point (a point of the labeled center point), which can be a sub-pixel point; x, y are points within the sigma range near the labeled point.

[0085] In addition, in the embodiment of the application, when calculating the heat value, 3*sigma pixel points are extended in the up, down, left and right directions of the labeled point to create a rectangular region of interest (if the image boundary is encountered, it is truncated at the boundary), and all pixel points in the rectangular region participate in the calculation of the heat value. The closer the distance to the labeled point, the greater the possibility that it is the labeled point, and the greater the HeatValue calculated, and vice versa.

[0086] The two types of feature points of the parking space, the center point of the empty parking space and the center point of the occupied parking space, correspond to two deep learning calculation channels, and the neural network performs regression on the two types of feature points labeled by the sample image, and uses a mean square error (MSE) loss function to supervise the model.

[0087]

[0088] wherein m is the number of pixel points participating in the calculation of MSE, y i is the true value, and f(x i ) is the neural network prediction value.

[0089] For example, for a sample image, the true value of the heat map is generated on a 104*104*2 size matrix, '2' represents two channels, which represent the empty parking space heat map and the occupied parking space heat map in turn. The two types of feature points, the center point of the empty parking space and the center point of the occupied parking space, in the sample image are defined as hot spots through Gaussian distribution, and the neural network performs regression on the true value, and uses a mean square error (MSE) loss function to supervise the model.

[0090] Referring to Figure 3The neural network prediction schematic diagram is shown. The image is input into the neural network, and the neural network outputs an empty parking space heat map and an occupied parking space heat map. The network structure of the neural network is not limited in the embodiment of the application.

[0091] It also needs to be explained that for a sample image, the size of the RGB image is 416*416*3. In order to improve the robustness of the algorithm, the sample image can be subjected to data random enhancement operation in six aspects of translation, rotation, hue, saturation, brightness, and contrast. The sample image is input into the neural network (the input size is 416*416*3, and the output size of the deep learning convolutional neural network is 104*104*2, that is, there is no special requirement for the network structure, and the specific network structure is not described here), and the output heat map size is 104*104*2. '2' represents 2 channels, and the empty parking space heat map and the occupied parking space heat map are sequentially represented.

[0092] In addition, in recent years, the deep learning technology has developed greatly in the field of visual perception. Through the data-driven algorithm model, the neural network can learn various visual environmental characteristics under data training, so as to provide the robustness of the algorithm in various scenes and greatly improve the performance of visual parking detection.

[0093] The data-driven algorithm idea needs a large amount of data as the basis of the algorithm, and provides various scenes to make the algorithm have better performance. Therefore, before the deep learning neural network starts training, the traditional method will perform data enhancement processing on the data, such as image translation, rotation, scaling, modifying image contrast, saturation, brightness, adding random noise, etc., so that the training required pictures have diversity. However, these data enhancement operations have always failed to solve the influence of different natural light on the scene, and cannot truly simulate the change of scene light.

[0094] The parking detection data collection work is generally a special data collection vehicle, which collects data in various parks, communities, basements and other functional places with parking spaces. The time period is the biggest influencing factor. However, most of the data collection is in the daytime, so that the data amount of the daytime scene accounts for a large proportion. The network model prediction result is very good in the daytime, but is poor at night. Because the outdoor light has obvious difference between the daytime and the night, the images collected by the camera also have obvious difference. For the deep learning neural network, the more diverse the data set used for training is, the better. However, the data collection work amount will increase by more than 2 times and the collection time will also increase by more than 2 times for the same community in different time periods, which has a great influence on the project development time.

[0095] Therefore, the data set of the current parking space detection algorithm based on the surround view stitching image, due to the fact that the scene on the ground is more common in the daytime during the data collection process, the result of the model in the night scene is not good, if completely relying on data collection, it will increase the data collection cost, data labeling cost, and lengthen the project development cycle. To this end, the present application can change the daytime data in the data set to the scene without street lights at night and with street lights at night, not only the change of image brightness, but also the influence of car lights, street lights and other lights on the whole image, generate night pictures that cannot be found by human eyes, expand the data set scene based on daytime data, reduce the data collection cost, shorten the development cycle and improve the development efficiency.

[0096] The following describes the acquisition method of the sample image in the embodiment of the present application, including the following steps:

[0097] Acquire a first daytime sample image;

[0098] A conversion model obtained by pre-training based on the generative adversarial network technology is called to convert the first daytime sample image into a first nighttime sample image, and the conversion model is obtained by training the generative adversarial network with the conversion result of the second daytime sample image tending to the second nighttime sample image labeled for the second daytime sample image as the target.

[0099] In the embodiment of the present application, the generative adversarial network is trained by the second daytime sample image and the second nighttime sample image to generate a conversion model. For the scene without street lights at night, the second nighttime sample image is the nighttime sample image when there is no street light at night; and for the scene with street lights at night, the second nighttime sample image is the nighttime sample image when there is street light at night. The number of second nighttime sample images in the two scenes is greater than 1000, and the number of second nighttime sample images between the two scenes is consistent. Of course, one conversion model can be trained for two scenes to obtain nighttime sample images in two scenes by converting the same daytime sample image. The training process of the conversion model is described in one scene in the embodiment of the present application, and the other scene is also the same.

[0100] Referring to Figure 4 The structure diagram of the generative adversarial network. The generative adversarial network includes two types of network structures, a generative network and a discriminative network. The role of the generative network is to generate a target style image using an input image, and the role of the discriminative network is to discriminate whether the image output by the generative network is similar to the target style.

[0101] Continuing to refer to Figure 4 The generative adversarial network in the embodiment of the present application includes a first generative network G A , a first discriminative network D A corresponding to the first generative network G B, the second generation network G B and the second discriminator network D corresponding to the second generation network A Correspondingly, the process of training the conversion model based on the generative adversarial network technology includes:

[0102] The second daytime sample image is input into the first generation network and the second discriminator network, and the second nighttime sample image is input into the first discriminator network and the second generation network, so as to:

[0103] The first generation network generates a third nighttime sample image based on the second daytime sample image and outputs it to the first discriminator network, so that the first discriminator network processes the second nighttime sample image and the third nighttime sample image; the second generation network generates a third daytime sample image based on the second nighttime sample image and outputs it to the second discriminator network, so that the second discriminator network processes the third daytime sample image and the second daytime sample image;

[0104] The first loss function value is calculated according to the processing result of the first discriminator network, and the second loss function value is calculated according to the processing result of the second discriminator network;

[0105] The network parameters of the first generation network, the second generation network, the first discriminator network and the second discriminator network are adjusted according to the first loss function value and the second loss function value respectively;

[0106] In the case that the corresponding training end condition is met, the adjusted first generation network is taken as the conversion model.

[0107] Suppose that the second daytime sample image is X, the second nighttime sample image is an image in a nighttime scene without street lamps on the ground, and Y. In order to generate nighttime data by using daytime parking data, X is input into the first generation network G A , the first generation network G A outputs G A (X), it is necessary to judge whether the output image G A (X) of the first generation network G A meets the scene requirements, and it is necessary to input it into the first discriminator network D B , the first discriminator network D B processes the input Y and G A (X) to complete the discrimination, and the discrimination index of the first discriminator network D B is as follows:

[0108]

[0109] In addition, Y is input into the second generation network G B , the second generation network G B outputs G B(Y), it is needed to judge whether the output image G B of the second generation network G B (Y) meets the scene requirement, and it is needed to input it into the second discrimination network D A , the second discrimination network D A processes the input X and G B (Y) to complete discrimination, and the discrimination index of the second discrimination network D A is the following adversarial loss function:

[0110]

[0111] Therefore, the first loss function value can be calculated according to the above formula (3), and the second loss function value can be calculated according to the above formula (4), and in the embodiment of the application, the sum of the first loss function value and the second loss function value is taken as the loss function value of the generative adversarial network, and then the network parameters of each network structure in the generative adversarial network are adjusted by using the gradient descent algorithm to converge the loss function value of the generative adversarial network.

[0112] In addition, the training end condition can be set according to the training time length or the training times, and the embodiment of the application does not limit this.

[0113] Further, in order to improve the performance of the conversion model, the embodiment of the application further considers:

[0114] If the daytime data can generate the nighttime data, and the nighttime data can also generate the daytime data, theoretically:

[0115] G B (G A (x))≈x (5)

[0116] G A (G B (y))≈y (6)

[0117] For this, the application adds a cycle Loss auxiliary neural network training:

[0118]

[0119] Therefore, in the embodiment of the application, the total LOSS of the generative adversarial network is:

[0120] LOSS=Loss GAN (G A , D A , X, Y)+Loss GAN (G B , D B X, Y)+γ*Loss oyc (G A , GB ) (8)

[0121] Wherein, gamma is a preset weight coefficient.

[0122] Based on this, the embodiment of the application further includes the following steps:

[0123] The third night sample image is input into the second generation network, and the third daytime sample image is input into the first generation network, so as to achieve:

[0124] The second generation network generates a fourth daytime sample image based on the third night sample image; and the first generation network generates a fourth night sample image based on the third daytime sample image.

[0125] According to the comparison results of the second daytime sample image and the fourth daytime sample image, and the comparison results of the second night sample image and the fourth night sample image, a third loss function value is calculated.

[0126] Accordingly, the network parameters of the first generation network, the second generation network, the first discriminator network and the second discriminator network are adjusted according to the first loss function value and the second loss function value, including:

[0127] The network parameters of the first generation network, the second generation network, the first discriminator network and the second discriminator network are adjusted according to the first loss function value, the second loss function value and the third loss function value.

[0128] In the embodiment of the application, the third loss function value is calculated according to the above formula (7), and then the sum of the first loss function value, the second loss function value and the third loss function value is taken as the loss function value of the generative adversarial network according to the above formula (8), and then the network parameters of each network structure in the generative adversarial network are adjusted by using the gradient descent algorithm to converge the loss function value of the generative adversarial network.

[0129] S30, target pixel points whose heat values meet corresponding threshold values are extracted from the empty parking space heat map and the occupied parking space heat map, and whether the parking space centered on the target pixel point is occupied is determined according to the heat value of the target pixel point.

[0130] In the embodiment of the application, continuing to refer to the neural network prediction schematic diagram shown in Figure 3 For the empty parking space heat map and the occupied parking space heat map output by the detection model, target pixel points whose heat values meet respective threshold values are extracted from the two heat maps.

[0131] Assuming that the threshold corresponding to the empty parking space heat map is 0.55, the heat values of different pixel points in the empty parking space heat map can be sorted from small to large, the pixel points with heat values less than 0.55 are removed, and for the remaining pixel points, the heat values are compared with the adjacent pixel points within a certain range of position distance, and the pixel point with the maximum heat value is retained. Then, the K pixel points with the maximum heat values are selected as the first target pixel points corresponding to the empty parking space heat map. Similarly, the K second target pixel points corresponding to the occupied parking space heat map can be obtained according to the threshold corresponding to the occupied parking space heat map. K is the maximum number of detected parking spaces. Continue to refer to Figure 3 , the pixel point 11 is the first target pixel point, and the pixel points 12-19 are the second target pixel points.

[0132] Therefore, for the first target pixel point, the heat value thereof is the probability that the parking space with the first target pixel point as the center point is an empty parking space. For the second target pixel point, the heat value thereof is the probability that the parking space with the second target pixel point as the center point is an occupied parking space.

[0133] In addition, it is also possible that a certain pixel point belongs to both the first target pixel point and the second target pixel point, that is, a certain target pixel point is located in both the empty parking space heat map and the occupied parking space heat map. Then, the first heat value of the target pixel point in the empty parking space heat map and the second heat value of the target pixel point in the occupied parking space heat map can be obtained, the maximum heat value is selected as the heat value with the maximum confidence, and the other heat value is discarded. The heat value is used to determine whether the parking space with the target pixel point as the center point is an empty parking space. For example, the heat value with the maximum confidence is the second heat value, which can be used as the probability that the parking space with the target pixel point as the center point is an occupied parking space.

[0134] In summary, it can be determined whether each parking space in the target image is an empty parking space or an occupied parking space.

[0135] The present application is based on the feature detection of whether the parking space can be parked based on the surround view spliced bird's eye view, and has the following advantages:

[0136] 1. It can be integrated into most current parking space detection algorithms, and the computing power increases very little.

[0137] 2. By judging the feature difference between the empty parking space and the unparkable parking space, it can be judged whether the parking space can be parked, which not only solves the problem that the target detection algorithm cannot estimate the physical distance of the intelligent detection of a specific target, but also solves the problem that the semantic segmentation drivable area detection has low reliability and cannot judge the opening and closing state of the ground lock.

[0138] 3. The detection accuracy is very high, and the accuracy in real vehicle test is greater than 99%.

[0139] Based on the parking space detection method provided in the above embodiments, the embodiment of the present application correspondingly provides a device for executing the above parking space detection method, a structure diagram of the device is shown in Figure 5 as follows, which comprises:

[0140] An image acquisition module 10 is configured to acquire a target image to be detected collected by a vehicle-mounted camera.

[0141] A heat map extraction module 20 is configured to extract an empty parking space heat map and an occupied parking space heat map of the target image, the empty parking space heat map containing heat values of different pixel points belonging to the center points of empty parking spaces, and the occupied parking space heat map containing heat values of different pixel points belonging to the center points of occupied parking spaces.

[0142] A parking space detection module 30 is configured to extract target pixel points with heat values satisfying corresponding threshold values from the empty parking space heat map and the occupied parking space heat map respectively, and determine whether a parking space with the target pixel point as the center point is occupied according to the heat value of the target pixel point.

[0143] Optionally, the heat map extraction module 20 is specifically configured to:

[0144] retrieve a detection model obtained by pre-training, the detection model being obtained by training a neural network with a goal that a prediction result of an empty parking space heat map of a sample image approaches a label result of the empty parking space heat map of the sample image, and a prediction result of an occupied parking space heat map of the sample image approaches a label result of the occupied parking space heat map of the sample image; and input the target image into the detection model to obtain the empty parking space heat map and the occupied parking space heat map output by the detection model for the target image.

[0145] Optionally, the heat map extraction module 20 acquires the sample image in the following manner:

[0146] acquire a first daytime sample image; retrieve a conversion model obtained by pre-training based on a generative adversarial network technology, and convert the first daytime sample image into a first nighttime sample image through the conversion model, the conversion model being obtained by training the generative adversarial network with a goal that a conversion result of a second daytime sample image approaches a second nighttime sample image labeled for the second daytime sample image.

[0147] Optionally, the generative adversarial network comprises a first generation network, a first discrimination network corresponding to the first generation network, a second generation network, and a second discrimination network corresponding to the second generation network.

[0148] Correspondingly, the process of training the conversion model by the heat map extraction module 20 based on the generative adversarial network technology comprises:

[0149] The second daytime sample image is input into the first generation network and the second discrimination network respectively, and the second nighttime sample image is input into the first discrimination network and the second generation network, so as to realize that the first generation network generates a third nighttime sample image based on the second daytime sample image and outputs to the first discrimination network, so that the first discrimination network processes the second nighttime sample image and the third nighttime sample image; the second generation network generates a third daytime sample image based on the second nighttime sample image and outputs to the second discrimination network, so that the second discrimination network processes the third daytime sample image and the second daytime sample image; a first loss function value is calculated according to the processing result of the first discrimination network, and a second loss function value is calculated according to the processing result of the second discrimination network; the network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network are adjusted according to the first loss function value and the second loss function value respectively; and the adjusted first generation network is taken as the conversion model when the corresponding training end condition is met.

[0150] Optionally, the heat map extraction module 20 is further configured to:

[0151] The third nighttime sample image is input into the second generation network, and the third daytime sample image is input into the first generation network, so as to realize that the second generation network generates a fourth daytime sample image based on the third nighttime sample image; the first generation network generates a fourth nighttime sample image based on the third daytime sample image; and a third loss function value is calculated according to the comparison result of the second daytime sample image and the fourth daytime sample image, and the comparison result of the second nighttime sample image and the fourth nighttime sample image.

[0152] Correspondingly, the heat map extraction module 20 adjusts the network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network according to the first loss function value and the second loss function value, including:

[0153] The network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network are adjusted according to the first loss function value, the second loss function value and the third loss function value.

[0154] Optionally, the parking space detection module 30 determines whether the parking space centered on the target pixel point is occupied according to the heat value of the target pixel point, including:

[0155] If the target pixel point is located in both the empty parking space heat map and the occupied parking space heat map, a first heat value of the target pixel point in the empty parking space heat map and a second heat value of the target pixel point in the occupied parking space heat map are obtained respectively; and the maximum heat value is selected from the first heat value and the second heat value as the heat value of the target pixel point.

[0156] Based on the parking space detection method provided in the above embodiments, the embodiment of the present application further provides an electronic device, which comprises at least one memory and at least one processor; the memory stores a program, and the processor invokes the program stored in the memory, and the program is used to realize the parking space detection method.

[0157] Based on the parking space detection method provided in the above embodiments, the embodiment of the present application further provides a storage medium, which stores computer executable instructions, and the computer executable instructions are used to execute the parking space detection method.

[0158] The above describes in detail the parking space detection method, device, electronic device and storage medium provided by the present application. The principle and implementation mode of the present application are described by using specific examples. The above embodiment is only used to help understand the method and core idea of the present application. Meanwhile, for the general technical personnel in the art, the specific implementation mode and application range will be changed according to the idea of the present application. Therefore, the content of the specification should not be understood as the limitation of the present application.

[0159] It should be noted that each embodiment in the specification adopts a progressive description mode, and each embodiment mainly describes the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the related part can be referred to the method part.

[0160] It should be further noted that, in this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device inherent to the process, method, article or device includes or further includes the inherent elements of the process, method, article or device. Without more limitation, the element defined by the statement "including a" does not exclude the existence of other same elements in the process, method, article or device including the element.

[0161] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of detecting a parkable parking space, characterized by, The method comprises: acquiring a target image to be detected collected by a vehicle-mounted camera; extracting an empty parking space heat map and an occupied parking space heat map of the target image, the empty parking space heat map containing heat values of different pixel points belonging to the center points of empty parking spaces, and the occupied parking space heat map containing heat values of different pixel points belonging to the center points of occupied parking spaces, the occupied parking spaces including parking spaces occupied by obstacles and parking spaces with ground locks in an open state; extracting target pixel points with heat values satisfying corresponding threshold values from the empty parking space heat map and the occupied parking space heat map, respectively, the heat value corresponding to the target pixel point of the empty parking space heat map representing the probability that the parking space with the target pixel point of the empty parking space heat map as the center point is an empty parking space, and the heat value corresponding to the target pixel point of the occupied parking space heat map representing the probability that the parking space with the target pixel point of the occupied parking space heat map as the center point is an occupied parking space; if the target pixel point is located in both the empty parking space heat map and the occupied parking space heat map, acquiring a first heat value of the target pixel point in the empty parking space heat map and a second heat value of the target pixel point in the occupied parking space heat map, respectively; and selecting the maximum heat value from the first heat value and the second heat value as the heat value of the target pixel point to determine whether the parking space with the target pixel point as the center point is occupied.

2. The method of claim 1, wherein, The extraction of the empty parking space heat map and the occupied parking space heat map of the target image comprises: calling a detection model obtained by pre-training, the detection model being obtained by training a neural network with the goal that the prediction result of the empty parking space heat map of a sample image approaches the label result of the empty parking space heat map of the sample image and the prediction result of the occupied parking space heat map of the sample image approaches the label result of the occupied parking space heat map of the sample image; inputting the target image into the detection model to acquire the empty parking space heat map and the occupied parking space heat map output by the detection model for the target image.

3. The method of claim 2, wherein, The acquisition of the sample image comprises: acquiring a first daytime sample image; calling a conversion model obtained by pre-training based on a generative adversarial network technology to convert the first daytime sample image into a first nighttime sample image through the conversion model, the conversion model being obtained by training a generative adversarial network with the goal that the conversion result of a second daytime sample image approaches the second nighttime sample image labeled for the second daytime sample image.

4. The method of claim 3, wherein, The generative adversarial network comprises a first generative network, a first discriminative network corresponding to the first generative network, a second generative network, and a second discriminative network corresponding to the second generative network. Correspondingly, the process of training the conversion model based on the generative adversarial network technology comprises: inputting the second daytime sample image into the first generative network and the second discriminative network, respectively, and inputting the second nighttime sample image into the first discriminative network and the second generative network, to realize The first generation network generates a third daytime sample image based on the second nighttime sample image, and outputs to the first discrimination network, so that the first discrimination network processes the second nighttime sample image and the third nighttime sample image; the second generation network generates a third nighttime sample image based on the second daytime sample image, and outputs to the second discrimination network, so that the second discrimination network processes the third daytime sample image and the second daytime sample image; According to the processing result of the first discrimination network, a first loss function value is calculated, and according to the processing result of the second discrimination network, a second loss function value is calculated; According to the first loss function value and the second loss function value, the network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network are adjusted respectively; In the case that the corresponding training end condition is met, the adjusted first generation network is taken as the conversion model.

5. The method of claim 4, wherein, The method further comprises: The third nighttime sample image is input to the second generation network, and the third daytime sample image is input to the first generation network, so as to: The second generation network generates a fourth daytime sample image based on the third nighttime sample image; the first generation network generates the fourth nighttime sample image based on the third daytime sample image; According to the comparison results of the second daytime sample image and the fourth daytime sample image, and the comparison results of the second nighttime sample image and the fourth nighttime sample image, a third loss function value is calculated; Correspondingly, the first loss function value and the second loss function value are used to adjust the network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network respectively, which comprises: The first loss function value, the second loss function value and the third loss function value are used to adjust the network parameters of the first generation network, the second generation network, the first discrimination network and the second discrimination network respectively.

6. A parkable space detection device, characterized by The device comprises: An image acquisition module is configured to acquire a target image to be detected collected by a vehicle-mounted camera; A heat map extraction module is configured to extract an empty parking space heat map and an occupied parking space heat map of the target image, wherein the empty parking space heat map contains heat values of different pixel points belonging to the center points of empty parking spaces, and the occupied parking space heat map contains heat values of different pixel points belonging to the center points of occupied parking spaces, and the occupied parking spaces include parking spaces occupied by obstacles and parking spaces with ground locks in an open state; The parking space detection module is configured to extract target pixel points with heat values satisfying corresponding threshold values from the empty parking space heat map and the occupied parking space heat map, respectively. The heat value corresponding to the target pixel point of the empty parking space heat map represents a probability that a parking space centered on the target pixel point of the empty parking space heat map is an empty parking space. The heat value corresponding to the target pixel point of the occupied parking space heat map represents a probability that a parking space centered on the target pixel point of the occupied parking space heat map is an occupied parking space. If the target pixel point is located in both the empty parking space heat map and the occupied parking space heat map, a first heat value of the target pixel point in the empty parking space heat map and a second heat value of the target pixel point in the occupied parking space heat map are obtained, respectively. The maximum heat value is selected from the first heat value and the second heat value as the heat value of the target pixel point to determine whether a parking space centered on the target pixel point is occupied.

7. The apparatus of claim 6, wherein, The heat map extraction module is specifically configured to: retrieve a detection model obtained by pre-training, the detection model being obtained by training a neural network with a goal that a prediction result of an empty parking space heat map of a sample image approaches a label result of the empty parking space heat map of the sample image and a prediction result of an occupied parking space heat map of the sample image approaches a label result of the occupied parking space heat map of the sample image; input the target image into the detection model to obtain an empty parking space heat map and an occupied parking space heat map output by the detection model for the target image.

8. An electronic device, comprising: The electronic device includes at least one memory and at least one processor. The memory stores a program. The processor invokes the program stored in the memory. The program is configured to implement the parkable parking space detection method of any one of claims 1-5.

9. A storage medium, characterized by The storage medium stores computer executable instructions. The computer executable instructions are configured to execute the parkable parking space detection method of any one of claims 1-5.

Images