Parking space detection method, device and system, vehicle and readable storage medium

Abstract

The invention relates to the technical field of parking space detection, and discloses a parking space detection method, device and system, a vehicle and a readable storage medium, and the method comprises the steps: obtaining a parking environment image, employing a neural network capable of carrying out the down-sampling and feature extraction of the parking environment image to extract the parking environment image, obtaining an image feature map, and obtaining a parking space feature map; converting the image feature map to obtain an environment feature map under a bird's-eye view by using a coordinate system conversion relation from image pixels to a vehicle, inputting the environment feature map under the bird's-eye view into a second feature extraction model to obtain a parking space feature map under the bird's-eye view, and detecting and positioning the parking space feature map under the bird's-eye view to obtain a parking space feature map under the bird's-eye view. The deep learning technology is applied to parking space detection under the BEV view angle, the accuracy of parking space detection is further improved, BEV space projection is only carried out on the down-sampled feature map, the detection is more efficient, and it is guaranteed that time consumed by vehicle end algorithm deployment is within a controllable range.

Description

Technical Field

[0001] The present invention relates to the technical field of parking space detection, and in particular to a parking space detection method, device, system, vehicle and readable storage medium. Background Art

[0002] With the rapid development of autonomous driving technology, parking space detection, as an important part of autonomous parking for autonomous vehicles, has become increasingly important for improving user experience and driving safety. Traditional parking space detection methods rely heavily on matching local features of parking spaces, such as parking space lines and angles. In complex environments, these local features are easily affected by factors such as lighting and occlusion, which affects the accuracy and stability of parking space detection. Summary of the Invention

[0003] In view of this, the present invention provides a parking space detection method, device, system, vehicle and readable storage medium to solve the problem that local features are easily interfered with by factors such as lighting and occlusion, affecting the accuracy and stability of parking space detection.

[0004] In a first aspect, the present invention provides a parking space detection method, the method comprising: acquiring a parking environment image captured by a vehicle-mounted image acquisition device; inputting the parking environment image into a first feature extraction model to obtain an image feature map, the first feature extraction model comprising a first neural network capable of downsampling and extracting features from the parking environment image; utilizing a preset image pixel to vehicle coordinate system conversion relationship to convert the image feature map into an environment feature map under a bird's-eye view; inputting the environment feature map under a bird's-eye view into a second feature extraction model to obtain a parking space feature map under a bird's-eye view, the second feature extraction model comprising a second neural network capable of extracting target parking space features from the environment feature map; and detecting and positioning the parking space feature map under a bird's-eye view to obtain a parking space detection result.

[0005] The parking space detection method provided by the present invention obtains a parking environment image, extracts the parking environment image using a first feature extraction model including a first neural network that can downsample and extract features of the parking environment image, and obtains an image feature map. The image feature map is then converted into an environment feature map under a bird's-eye view using a conversion relationship from image pixels to the coordinate system of the vehicle, and the environment feature map under the bird's-eye view is input into a second feature extraction model to obtain a parking space feature map under a bird's-eye view. The parking space feature map under the bird's-eye view is detected and positioned to obtain a parking space detection result. The method realizes the application of deep learning technology to parking space detection from a BEV perspective, and recognizes parking space features by training a deep learning model to further improve the accuracy of parking space detection. The method only projects the downsampled feature map into the BEV space, making the detection more efficient and ensuring that the time consumed for vehicle-side algorithm deployment is within a controllable range.

[0006] In an optional embodiment, the image feature map is converted into an environment feature map under a bird's-eye view by using a preset image pixel to vehicle coordinate system conversion relationship, including: using the focal length and center point coordinates of the vehicle-mounted image acquisition device to determine the projection matrix of the image acquisition device, and multiplying the image feature map by the image acquisition device projection matrix to obtain the image feature map in the image acquisition device coordinate system; using the posture relationship between the vehicle-mounted image acquisition device and the vehicle to determine the rotation matrix and translation matrix corresponding to the image acquisition device; based on the rotation matrix and translation matrix corresponding to the image acquisition device, the image feature map in the image acquisition device coordinate system and the vehicle coordinates, determining the environment feature map under the bird's-eye view.

[0007] The present invention realizes the conversion of the image coordinate system into the environmental characteristic map under the BEV space coordinate system, and then detects the parking space information based on the environmental characteristic map under the BEV space, aiming to improve the accuracy and real-time performance of parking space detection.

[0008] In an optional embodiment, the detecting and locating the parking space feature map under the bird's-eye view to obtain the parking space detection result includes: using a head network to detect and locate the parking space feature map under the bird's-eye view, and outputting the parking space detection result.

[0009] The present invention utilizes the head network to detect and locate the parking space feature map and outputs the detection result of the parking space, which can improve the accuracy of the parking space detection result.

[0010] In an optional embodiment, the head network includes at least a global detection network and a local detection network. The head network is used to detect and locate the parking space feature map under the bird's-eye view, and output the detection result of the parking space, including: using the global detection network to detect the parking space feature map under the bird's-eye view, and obtain the overall attribute information of the parking space and the first positions corresponding to the four corner points of the parking space, wherein the overall attribute information includes at least the occupancy status of the parking space, the parking space type and the parking space line direction; using the local detection network to detect the parking space feature map under the bird's-eye view, and obtain the second positions and parking space line angles corresponding to the four corner points of the parking space; based on the overall attribute information of the parking space, the first positions, the second positions and the parking space line angles corresponding to the four corner points, the parking space detection result is determined.

[0011] The present invention adopts a dual strategy of global detection network and local detection network to obtain accurate parking space information, and obtains corresponding accurately obtainable parking space information based on different focuses, which can improve the accuracy of parking space information.

[0012] In an optional embodiment, the method of determining the detection result of the parking space based on the overall attribute information of the parking space, the first position and the second position corresponding to the four corner points, and the parking space line angle includes: calculating the Euclidean distance between the first position of the current corner point of the parking space obtained by the global detection network and the second position of the same corner point obtained by the local detection network; judging whether the Euclidean distance is less than a preset Euclidean distance threshold; if the Euclidean distance is less than the preset Euclidean distance, updating the first position of the current corner point obtained by the global detection network to the second position of the current corner point obtained by the local detection network; determining the detection result of the parking space based on the overall attribute information of the parking space, the updated first positions corresponding to the four corner points, and the parking space line angle.

[0013] The present invention updates the corner point positions detected by the local detection network to the corner point position coordinates detected by the global detection network, which not only improves the accuracy of parking space corner point positioning, but also ensures the integrity and accuracy of parking space information, providing strong support for subsequent applications such as vehicle navigation, parking management and monitoring.

[0014] In an optional embodiment, the first neural network and the second neural network are both composed of a backbone network and a neck network, which are used for feature extraction processing.

[0015] The present invention adopts different backbone networks and neck networks for the first neural network and the second neural network respectively, and the corresponding network type can be set according to actual needs, making feature extraction more flexible and accurate.

[0016] In an optional embodiment, the vehicle-mounted image acquisition device is a fisheye camera. Before using a preset image pixel to vehicle coordinate system conversion relationship to convert the image feature map to obtain an environmental feature map under a bird's-eye view, the method also includes: using the distortion parameters of the fisheye camera to calibrate the image feature map.

[0017] The present invention uses the distortion parameters corresponding to the fisheye camera installed on the vehicle to calibrate the image feature map, which can eliminate the barrel distortion unique to the fisheye camera and thus improve the accuracy of subsequent parking space detection.

[0018] In an optional embodiment, the first feature extraction model and the second feature extraction model are trained through the following steps: obtaining a training set of parking environment images captured by the vehicle-mounted image acquisition device and actual parking space information corresponding to the parking environment images; performing data augmentation processing on the parking environment image training set, wherein the data augmentation processing includes at least one of random cropping, scale change, and horizontal and vertical flipping; and optimizing the first feature extraction model and the second feature extraction model using the parking environment image training set after data augmentation processing and the actual parking space information corresponding to the parking environment images to obtain optimized first feature extraction models and second feature extraction models.

[0019] The present invention improves the accuracy of parking space detection by performing data enhancement processing on the environmental image training set, using the image data after data enhancement processing to train and optimize the model, and finally using the optimized feature extraction mode to execute the parking space detection method.

[0020] In a second aspect, the present invention provides a parking space detection device, comprising: an image capture module for acquiring a parking environment image captured by a vehicle-mounted image acquisition device; a feature map acquisition module for inputting the parking environment image into a first feature extraction model to obtain an image feature map, wherein the first feature extraction model includes a first neural network capable of downsampling and extracting features from the parking environment image; a feature map conversion module for converting the image feature map into an environment feature map under a bird's-eye view using a preset image pixel to vehicle coordinate system conversion relationship; a parking space feature extraction module for inputting the environment feature map under a bird's-eye view into a second feature extraction model to obtain a parking space feature map under a bird's-eye view, wherein the second feature extraction model includes a second neural network capable of extracting target parking space features from the environment feature map; and a parking space information detection module for detecting and locating the parking space feature map under the bird's-eye view to obtain a parking space detection result.

[0021] In a third aspect, the present invention provides a parking space detection system, which includes a controller, and the controller includes: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the parking space detection method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0022] In a fourth aspect, the present invention provides a vehicle comprising a parking space detection system.

[0023] In an optional embodiment, the vehicle further includes a fisheye camera mounted on the vehicle.

[0024] In a fifth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the parking space detection method of the first aspect or any corresponding embodiment thereof.

[0025] The parking space detection method provided by the present invention obtains a parking environment image, extracts the parking environment image using a first feature extraction model including a first neural network that can downsample and extract features of the parking environment image, and obtains an image feature map. The image feature map is then converted into an environment feature map under a bird's-eye view using a conversion relationship from image pixels to the coordinate system of the vehicle, and the environment feature map under the bird's-eye view is input into a second feature extraction model to obtain a parking space feature map under a bird's-eye view. The parking space feature map under the bird's-eye view is detected and positioned to obtain a parking space detection result. The method realizes the application of deep learning technology to parking space detection from a BEV perspective, and recognizes parking space features by training a deep learning model to further improve the accuracy of parking space detection. The method only projects the downsampled feature map into the BEV space, making the detection more efficient and ensuring that the time consumed for vehicle-side algorithm deployment is within a controllable range. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 is a flow chart of a parking space detection method according to an embodiment of the present invention;

[0028] Figure 2 is a flow chart of another parking space detection method according to an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of installing a fisheye camera according to an embodiment of the present invention;

[0030] Figure 4 2. It is an example diagram of parking space detection results output by a local detection network and a global detection network according to an embodiment of the present invention;

[0031] Figure 5 is an example diagram of a process for parking space detection according to an embodiment of the present invention;

[0032] Figure 6 is a structural block diagram of a parking space detection device according to an embodiment of the present invention;

[0033] Figure 7is a structural block diagram of a parking space detection system according to an embodiment of the present invention;

[0034] Figure 8 is a structural block diagram of a vehicle according to an embodiment of the present invention;

[0035] Figure 9 Schematic diagram of the hardware structure of the controller according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0037] According to an embodiment of the present invention, an embodiment of a parking space detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] In this embodiment, a parking space detection method is provided, which can be used in the controller of the above parking space detection system. Figure 1 FIG. 1 is a flow chart of a parking space detection method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0039] Step S101: Acquire a parking environment image captured by a vehicle-mounted image acquisition device.

[0040] The embodiments of the present invention do not limit the vehicle-mounted image acquisition device and can be used to capture images based on the image acquisition device actually installed on the vehicle, such as a camera, image sensor, etc., and there is no limit on the number of vehicle-mounted image acquisition devices installed on the vehicle. Generally, in order to achieve image acquisition in a surround-view scene, four image acquisition devices can be installed. As an example only, a real-value vehicle can be used to capture image data in a surround-view parking scene, and the parking environment images captured by the image acquisition device can be captured while ensuring timestamp synchronization.

[0041] Step S102: Input the parking environment image into a first feature extraction model to obtain an image feature map.

[0042] The first feature extraction model includes a first neural network that can downsample and extract features from the parking environment image.

[0043] In an embodiment of the present invention, a first feature extraction model can be used to perform two-dimensional feature extraction on a parking environment image. The first feature extraction model includes a first neural network capable of downsampling and extracting features from the parking environment image, such as a convolutional neural network. This neural network has the characteristics of local perception, parameter sharing, and downsampling, thereby improving the efficiency of image processing and feature extraction. The parking environment image can then be input into the first feature extraction model to obtain a downsampled feature map.

[0044] In step S103 , the image feature map is converted into an environment feature map under a bird's-eye view using a preset image pixel to vehicle coordinate system conversion relationship.

[0045] After obtaining the downsampled feature map, the embodiment of the present invention can convert the downsampled image feature map from pixel coordinates into an environmental feature map under a bird's eye view (BEV) based on a preset image pixel to vehicle coordinate system conversion relationship. It can also convert the two-dimensional downsampled feature map into an environmental feature map under a BEV space through a multi-layer perceptron (MLP) algorithm.

[0046] Step S104: input the environment feature map under the bird's-eye view into the second feature extraction model to obtain the parking space feature map under the bird's-eye view.

[0047] The second feature extraction model includes a second neural network that can extract target parking space features from the environmental feature map.

[0048] The second feature extraction model designed in the embodiment of the present invention includes a second neural network that can extract target parking space features from the environmental feature map, so as to accurately extract the target parking space features. The second neural network is not limited. For example, the DeepSort feature extraction network is an algorithm for target tracking, which can detect the target parking space features in the environmental feature map and obtain the parking space feature map in the BEV space. This is only an example.

[0049] Step S105 , detecting and positioning the parking space feature map under the bird's-eye view to obtain a detection result of the parking space.

[0050] The embodiment of the present invention can use traditional detection segmentation maps, parking space set results and other methods to detect and locate the parking space feature map under the bird's-eye view to obtain the detection results of the parking space. The detection results of the parking space may include parking space location information, parking space type, parking space occupancy status, etc., as an example only.

[0051] The parking space detection method provided in this embodiment obtains a parking environment image, extracts the parking environment image using a first feature extraction model including a first neural network that can downsample and extract features of the parking environment image, and obtains an image feature map. The image feature map is then converted into an environment feature map under a bird's-eye view using a conversion relationship from image pixels to the vehicle's coordinate system. The environment feature map under the bird's-eye view is input into a second feature extraction model to obtain a parking space feature map under a bird's-eye view. The parking space feature map under the bird's-eye view is detected and located to obtain a parking space detection result. This realizes the application of deep learning technology to parking space detection from the BEV perspective. By training a deep learning model to identify parking space features, the accuracy of parking space detection is further improved. Only the downsampled feature map is projected into the BEV space, making detection more efficient and ensuring that the time spent on vehicle-side algorithm deployment is within a controllable range.

[0052] In this embodiment, a parking space detection method is provided, which can be used in a controller in a parking space detection system. Figure 2 FIG. 1 is a flow chart of a parking space detection method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0053] Step S201: Acquire a parking environment image captured by a vehicle-mounted image acquisition device.

[0054] In the embodiment of the present invention, fisheye cameras can be installed in the front, left, right and rear of the real car respectively. The specific installation method is as follows: Figure 3 As shown, under the premise of ensuring timestamp synchronization, image data captured by four fisheye cameras are collected, wherein the resolution of the image captured by the fisheye camera is generally 1920×1080, which is only used as an example.

[0055] Step S202: Input the parking environment image into a first feature extraction model to obtain an image feature map.

[0056] The first feature extraction model includes a first neural network that can downsample and extract features from the parking environment image. Specifically, the first neural network consists of a backbone network and a neck network for feature extraction.

[0057] In an embodiment of the present invention, MobilenetV2 can be designed as a backbone network to downsample the parking environment image and extract two-dimensional features. The neck network can use BiFPN to further process and integrate the features extracted from the backbone network, such as feature fusion, feature dimensionality reduction, and feature enhancement, and can output a 4x downsampled image feature map, as an example only.

[0058] In an optional embodiment, the vehicle-mounted image acquisition device is a fisheye camera. Before using a preset image pixel to vehicle coordinate system conversion relationship to convert the image feature map into an environmental feature map under a bird's-eye view, the image feature map needs to be calibrated using the distortion parameters of the fisheye camera.

[0059] After obtaining a 4x downsampled image feature map, the embodiment of the present invention can calibrate the image feature map using the distortion parameters corresponding to the fisheye camera installed on the vehicle. This ensures that the straight lines in the image feature map remain straight lines after correction, eliminating the barrel distortion unique to fisheye cameras and thereby improving the accuracy of subsequent parking space detection.

[0060] In step S203 , the image feature map is converted into an environment feature map under a bird's-eye view using a preset image pixel to vehicle coordinate system conversion relationship.

[0061] Specifically, the above step S203 includes:

[0062] Step S2031: determine the image acquisition device projection matrix using the focal length and center point coordinates of the vehicle-mounted image acquisition device, and multiply the image feature map by the image acquisition device projection matrix to obtain the image feature map in the image acquisition device coordinate system.

[0063] Step S2032: using the position relationship between the vehicle-mounted image acquisition device and the vehicle, determine the rotation matrix and translation matrix corresponding to the image acquisition device.

[0064] Step S2033: Determine the environment feature map under the bird's-eye view based on the rotation matrix and translation matrix corresponding to the image acquisition device, the image feature map under the image acquisition device coordinate system, and the vehicle coordinates.

[0065] In an embodiment of the present invention, an inverse perspective mapping (IPM) module is used to perform matrix transformation on the two-dimensional image features extracted from the surround fisheye image, the corresponding camera distortion coefficients and the internal and external parameters, to obtain a transformation matrix from the image coordinate system to the vehicle coordinate system, thereby realizing the projection of the image to the BEV space. The camera internal parameters include parameters such as the focal length and the principal point coordinates, i.e., the center point coordinates. These parameters describe the relationship between the camera coordinate system and the image coordinate system. The camera matrix K is obtained through the focal length and the center point coordinates, K = [[fx, 0, cx], [0, fy, cy], [0, 0, 1]], where K is the perspective projection matrix of the camera, fx and fy are the focal lengths of the camera in the x and y directions, respectively, and cx and cy are the center points of the image. The calibrated image feature map can be multiplied by the perspective projection matrix of the camera to obtain the image feature map in the camera coordinate system.

[0066] The camera extrinsics of the embodiment of the present invention describe the conversion relationship between the camera coordinate system and the vehicle coordinate system (VCS). The rotation matrix and translation vector are determined based on the relative pose relationship between the camera and the vehicle. The camera coordinate system is converted to the vehicle VCS coordinate system using the following formula:

[0067]

[0068] Among them, (x c ,y c ,z c ) is the camera coordinate, R is the rotation matrix in the external parameter, T is the translation matrix in the external parameter, (x w ,y w ,z w ) are the corresponding ego-vehicle coordinates.

[0069] In the embodiment of the present invention, the image feature map in the camera coordinate system can be converted into the environment feature map in the bird's-eye view through the above formula.

[0070] The present invention realizes the conversion of the image coordinate system into the environmental characteristic map under the BEV space coordinate system, and then detects the parking space information based on the environmental characteristic map under the BEV space, aiming to improve the accuracy and real-time performance of parking space detection.

[0071] Step S204: input the environment feature map under the bird's-eye view into the second feature extraction model to obtain the parking space feature map under the bird's-eye view.

[0072] The second feature extraction model includes a second neural network that can extract target parking space features from the environmental feature map.

[0073] Specifically, the second neural network consists of a backbone network and a neck network, which are used to extract target parking space features.

[0074] In the embodiment of the present invention, VargNetV2 can be used as the backbone network, and UNet can be used as the neck network to extract target parking space features from the environmental feature map, thereby improving the generalization ability of the model and performing better target parking space detection tasks.

[0075] In an optional embodiment, the first feature extraction model and the second feature extraction model are trained through the following steps: obtaining a training set of parking environment images captured by a vehicle-mounted image acquisition device and actual parking space information corresponding to the parking environment images; performing data augmentation processing on the parking environment image training set, where the data augmentation processing includes at least one of random cropping, scale change, and horizontal and vertical flipping; and optimizing the first feature extraction model and the second feature extraction model using the parking environment image training set after data augmentation processing and the actual parking space information corresponding to the parking environment images to obtain optimized first feature extraction models and second feature extraction models.

[0076] In an embodiment of the present invention, a training set of parking environment images in a surround parking scenario is captured by a fisheye camera. Data enhancement processing can be performed on the training set of parking environment images, which may include random cropping, random scaling, random horizontal and vertical flipping, random application of affine transformation, and random color filling. Cropping can be performed by randomly cropping 28 pixels up and down and 96 pixels left and right, resulting in an image size of (768, 512). The enhanced training set of parking environment images can then be input into a first feature extraction model to obtain an image feature map, which is then converted into an environment feature map in a BEV space through an IPM module. A parking space feature map is then obtained through a second feature extraction module. Finally, the parking space feature map is detected to obtain a parking space information detection result. The parking space information detection result can be compared with the actual parking space information corresponding to the parking environment image. The first feature extraction model and the second feature extraction model can be optimized based on the difference results of the comparison. Ultimately, optimized and trained first and second feature extraction models can be obtained.

[0077] The present invention improves the accuracy of parking space detection by performing data enhancement processing on the environmental image training set, using the image data after data enhancement processing to train and optimize the model, and finally using the optimized feature extraction mode to execute the parking space detection method.

[0078] Step S205 , detecting and positioning the parking space feature map under the bird's-eye view to obtain a detection result of the parking space.

[0079] Specifically, the above step S205 includes:

[0080] In step S2051, the head network is used to detect and locate the parking space feature map under the bird's-eye view, and the detection result of the parking space is output.

[0081] Neural networks are usually divided into three parts: backbone network, neck network and head network. The backbone network is responsible for extracting features from the original input data, the neck network is used to further process and integrate the features extracted from the backbone network, and the head network is located after the neck network and is used to perform specific tasks such as classification, detection, regression and segmentation. It is responsible for converting the features extracted from the backbone network and the neck network into the final output.

[0082] After the embodiment of the present invention uses the second feature extraction module to obtain the parking space feature map, the parking space feature map can be detected by the detection head module, i.e., the head network, to calculate and output parking space information such as parking space category, occupancy status, and corner offset of the parking space. Among them, the head network is also a deep separable convolutional network structure. During the process of training the model, the detection head module can also be trained and optimized, just as an example.

[0083] Specifically, the head network includes at least a global detection network and a local detection network. The above step S2051 includes:

[0084] In step a1, a global detection network is used to detect a parking space feature map under a bird's-eye view to obtain overall attribute information of the parking space and first positions corresponding to the four corner points of the parking space.

[0085] The overall attribute information includes at least the occupancy status of the parking space, the parking space type, and the parking space line direction.

[0086] Step a2: Use the local detection network to detect the parking space feature map under the bird's-eye view to obtain the second positions and parking space line angles corresponding to the four corner points of the parking space.

[0087] Step a3: Determine a detection result of the parking space based on the overall attribute information of the parking space, the first position and the second position corresponding to the four corner points, and the parking space line angle.

[0088] In the process of parking space detection and positioning, the embodiment of the present invention can adopt a dual strategy to obtain detailed information of the parking space. A global detection network and a local detection network can be referenced. The global detection network is used to capture the overall attributes of the parking space, such as Figure 4 As shown, it may include but is not limited to the occupancy status of the parking space (i.e., whether there is a parked vehicle), the parking space type (such as vertical, parallel, oblique parking space, etc.), the confidence level (whether the above-mentioned overall attribute output is credible), the direction of the parking space line, and the first position corresponding to the four corner points of the parking space can also be obtained; the local detection network can be used to calculate the parking space feature map under the bird's-eye view, such as Figure 4 As shown, the first position, parking space type, parking space line angle and confidence level (used to indicate whether the above calculation output is credible) corresponding to the four corner points of the parking space are obtained, which is only for example.

[0089] When determining the detection results of parking spaces based on the global detection network and the local detection network, the embodiment of the method can select the pre-set attribute information that each network is mainly responsible for obtaining as the detection result of the parking space. For example, the more accurate information obtained by the global detection network includes occupancy status, parking space type, etc., and the more accurate information obtained by the local detection network includes the second position of the four corner points and the parking space line angle. The overall fusion outputs the detection result of the parking space; the parking space information output by two networks with higher confidence levels can also be selected as the detection result of the parking space, just as an example.

[0090] The present invention adopts a dual strategy of global detection network and local detection network to obtain accurate parking space information, and obtains corresponding accurately obtainable parking space information based on different focuses, which can improve the accuracy of parking space information.

[0091] Specifically, the above step a3 includes:

[0092] Step b1: calculating the Euclidean distance between a first position of a current corner point of a parking space obtained by the global detection network and a second position of the same corner point obtained by the local detection network.

[0093] Step b2: determine whether the Euclidean distance is less than a preset Euclidean distance threshold.

[0094] Step b3: If the Euclidean distance is less than the preset Euclidean distance, the first position of the current corner point obtained by the global detection network is updated to the second position of the current corner point obtained by the local detection network.

[0095] Step b4: determining a detection result of the parking space based on the overall attribute information of the parking space, the updated first positions corresponding to the four corner points, and the parking space line angle.

[0096] The local detection network designed in the embodiment of the present invention focuses on accurately estimating the position of the corner points of the parking space, because the accuracy of the corner points is crucial for applications such as vehicle parking, navigation and monitoring. After using the local head to obtain the second positions of the four corner points with high precision, the first position of the current corner point of the parking space obtained by the global detection network and the second position of the same corner point obtained by the local detection network can be calculated to perform Euclidean distance calculation. By setting a reasonable distance threshold, it is determined whether the Euclidean distance between the first position and the second position of the corner point is less than the preset distance threshold. If it is less than the distance threshold, it can be determined that the precise corner point obtained by the local detection network is effectively matched with the global detection network. The global detection network is updated with the position coordinates of the corner point detected by the local detection network, and parking space information containing precise corner point position information can be obtained.

[0097] The present invention updates the corner point positions detected by the local detection network to the corner point position coordinates detected by the global detection network, which not only improves the accuracy of parking space corner point positioning, but also ensures the integrity and accuracy of parking space information, providing strong support for subsequent applications such as vehicle navigation, parking management and monitoring.

[0098] In a specific embodiment, Figure 5 As shown, the overall process of the parking space detection method includes: inputting the parking environment image taken by the fisheye camera into the first feature extraction model, namely the image feature extraction network, to obtain a downsampled image feature map, and then using the internal and external parameters of the camera to convert the image feature map in the image pixel coordinate system into an environment feature map in the BEV space through the IPM module, and inputting the environment feature map in the BEV space into the second feature extraction model, namely the BEV space feature extraction network, to obtain a parking space feature map, and finally calculating and outputting the parking space detection result through the detection head module, namely the head network. For detailed description, please refer to the above embodiment and will not be repeated here.

[0099] This embodiment also provides a parking space detection device for implementing the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0100] This embodiment provides a parking space detection device, such as Figure 6 As shown, it includes: an image capturing module 601, which is used to acquire a parking environment image captured by a vehicle-mounted image acquisition device; a feature map acquisition module 602, which is used to input the parking environment image into a first feature extraction model to obtain an image feature map, wherein the first feature extraction model includes a first neural network that can downsample and extract features from the parking environment image; a feature map conversion module 603, which is used to convert the image feature map into an environment feature map under a bird's-eye view using a preset image pixel to vehicle coordinate system conversion relationship; a parking space feature extraction module 604, which is used to input the environment feature map under a bird's-eye view into a second feature extraction model to obtain a parking space feature map under a bird's-eye view, wherein the second feature extraction model includes a second neural network that can extract target parking space features from the environment feature map; and a parking space information detection module 605, which is used to detect and locate the parking space feature map under the bird's-eye view to obtain a parking space detection result.

[0101] In some optional embodiments, the feature map conversion module 603 includes: a camera coordinate system conversion unit, which is used to determine the projection matrix of the image acquisition device using the focal length and center point coordinates of the vehicle-mounted image acquisition device, and multiply the image feature map by the image acquisition device projection matrix to obtain the image feature map in the image acquisition device coordinate system; a matrix determination unit, which is used to determine the rotation matrix and translation matrix corresponding to the image acquisition device using the posture relationship between the vehicle-mounted image acquisition device and the vehicle; and a vehicle coordinate system conversion unit, which is used to determine the environment feature map under the bird's-eye view based on the rotation matrix and translation matrix corresponding to the image acquisition device, the image feature map in the image acquisition device coordinate system, and the vehicle coordinates.

[0102] In some optional implementations, the parking space information detection module 605 includes: a head network detection unit, configured to detect and locate the parking space feature map under the bird's-eye view using the head network, and output a detection result of the parking space.

[0103] In some optional embodiments, the head network includes at least a global detection network and a local detection network, and the parking space information detection module 605 includes: a global network detection unit, which is used to use the global detection network to detect the parking space feature map under the bird's-eye view, and obtain the overall attribute information of the parking space and the first positions corresponding to the four corner points of the parking space, and the overall attribute information includes at least the occupancy status of the parking space, the parking space type and the parking space line direction; a local network detection unit, which is used to use the local detection network to detect the parking space feature map under the bird's-eye view, and obtain the second positions and parking space line angles corresponding to the four corner points of the parking space; a detection result output unit, which is used to determine the detection result of the parking space based on the overall attribute information of the parking space, the first positions, the second positions and the parking space line angles corresponding to the four corner points.

[0104] In some optional embodiments, the detection result output unit includes: a Euclidean distance calculation subunit, used to calculate the Euclidean distance between the first position of the current corner point of the parking space obtained by the global detection network and the second position of the same corner point obtained by the local detection network; a distance judgment subunit, used to judge whether the Euclidean distance is less than a preset Euclidean distance threshold; a position update subunit, used to update the first position of the current corner point obtained by the global detection network to the second position of the current corner point obtained by the local detection network if the Euclidean distance is less than the preset Euclidean distance; a parking space detection result output subunit, used to determine the detection result of the parking space based on the overall attribute information of the parking space, the updated first positions corresponding to the four corner points, and the parking space line angle.

[0105] In some optional embodiments, the first neural network and the second neural network are both composed of a backbone network and a neck network for performing feature extraction processing.

[0106] In some optional embodiments, the vehicle-mounted image acquisition device is a fisheye camera. Before using a preset image pixel to vehicle coordinate system conversion relationship to convert the image feature map to obtain an environmental feature map under a bird's-eye view, the parking space detection device also includes: an image calibration module for calibrating the image feature map using the distortion parameters of the fisheye camera.

[0107] In some optional embodiments, the first feature extraction model and the second feature extraction model are trained through the following steps: obtaining a training set of parking environment images captured by an on-board image acquisition device and actual parking space information corresponding to the parking environment images; performing data enhancement processing on the parking environment image training set, where the data enhancement processing includes at least one of random cropping, scale change, and horizontal and vertical flipping; and optimizing the first feature extraction model and the second feature extraction model using the parking environment image training set after data enhancement processing and the actual parking space information corresponding to the parking environment images to obtain optimized first feature extraction models and second feature extraction models.

[0108] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0109] The parking space detection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0110] The embodiment of the present invention also provides a parking space detection system, such as Figure 7 As shown, the parking space detection system includes a controller, which includes a memory and a processor. The memory and the processor are communicatively connected to each other. The memory stores computer instructions, and the processor executes the parking space detection method performed in the above embodiment by executing the computer instructions.

[0111] The embodiment of the present invention further provides a vehicle, such as Figure 8 As shown, the vehicle includes Figure 7 The parking space detection system shown in FIG. The vehicle also includes a fisheye camera installed on the vehicle.

[0112] See also Figure 9 , Figure 9 : is a schematic diagram of the structure of a controller provided by an optional embodiment of the present invention, such as Figure 9As shown, the controller includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the controller, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple controllers can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 9 A processor 10 is taken as an example.

[0113] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0114] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0115] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the controller, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the controller via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0116] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0117] The controller also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 can be connected via a bus or other means. Figure 9The bus connection is taken as an example.

[0118] The input device 30 can receive input digital or character information and generate key signal input related to the user settings and function control of the controller, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0119] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0120] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A parking space detection method, characterized in that: The method comprises: Acquire a parking environment image captured by a vehicle-mounted image acquisition device; Inputting the parking environment image into a first feature extraction model to obtain an image feature map, wherein the first feature extraction model includes a first neural network capable of downsampling and extracting features from the parking environment image; The image feature map is converted into an environment feature map under a bird's-eye view by using a preset image pixel to vehicle coordinate system conversion relationship; Inputting the environment feature map under the bird's-eye view into a second feature extraction model to obtain a parking space feature map under the bird's-eye view, wherein the second feature extraction model includes a second neural network capable of extracting target parking space features from the environment feature map; The parking space feature map under the bird's-eye view is detected and positioned to obtain a detection result of the parking space.

2. The method according to claim 1, characterized in that The method of using a preset image pixel to vehicle coordinate system conversion relationship to convert the image feature map to obtain an environment feature map under a bird's-eye view includes: Determine the projection matrix of the image acquisition device using the focal length and center point coordinates of the vehicle-mounted image acquisition device, and multiply the image feature map by the projection matrix of the image acquisition device to obtain the image feature map in the image acquisition device coordinate system; Determine the rotation matrix and translation matrix corresponding to the image acquisition device by using the position relationship between the vehicle-mounted image acquisition device and the vehicle; Based on the rotation matrix, translation matrix, image feature map in the image acquisition device coordinate system and vehicle coordinates corresponding to the image acquisition device, an environment feature map in the bird's-eye view is determined.

3. The method according to claim 1, characterized in that: The detecting and locating the parking space feature map under the bird's-eye view to obtain the detection result of the parking space includes: The head network is used to detect and locate the parking space feature map under the bird's-eye view, and the detection result of the parking space is output.

4. The method according to claim 3, characterized in that The head network at least includes a global detection network and a local detection network. The head network is used to detect and locate the parking space feature map under the bird's-eye view, and output the detection result of the parking space, including: Using a global detection network to detect the parking space feature map under the bird's-eye view, obtain overall attribute information of the parking space and first positions corresponding to four corner points of the parking space, wherein the overall attribute information at least includes an occupancy state of the parking space, a parking space type, and a parking space line direction; Using a local detection network to detect the parking space feature map under the bird's-eye view, and obtain the second positions and parking space line angles corresponding to the four corner points of the parking space respectively; The detection result of the parking space is determined based on the overall attribute information of the parking space, the first position and the second position respectively corresponding to the four corner points, and the parking space line angle.

5. The method according to claim 4, characterized in that The determining of the detection result of the parking space based on the overall attribute information of the parking space, the first position, the second position and the parking space line angle respectively corresponding to the four corner points includes: Calculating the Euclidean distance between a first position of a current corner point of the parking space obtained by the global detection network and a second position of the same corner point obtained by the local detection network; Determining whether the Euclidean distance is less than a preset Euclidean distance threshold; If the Euclidean distance is less than a preset Euclidean distance, updating the first position of the current corner point obtained by the global detection network to the second position of the current corner point obtained by the local detection network; The detection result of the parking space is determined based on the overall attribute information of the parking space, the updated first positions corresponding to the four corner points, and the parking space line angle.

6. The method according to claim 1, characterized in that The first neural network and the second neural network are both composed of a backbone network and a neck network, and are used for feature extraction processing.

7. The method according to claim 1, characterized in that The vehicle-mounted image acquisition device is a fisheye camera. Before converting the image feature map to obtain an environment feature map under a bird's-eye view by using a preset image pixel to vehicle coordinate system conversion relationship, the method further includes: The image feature map is calibrated using the distortion parameters of the fisheye camera.

8. The method according to claim 1, characterized in that The first feature extraction model and the second feature extraction model are trained by the following steps: Acquire a parking environment image training set captured by the vehicle-mounted image acquisition device and actual parking space information corresponding to the parking environment image; Performing data enhancement processing on the parking environment image training set, wherein the data enhancement processing includes at least one of random cropping, scale change, and horizontal and vertical flipping; The first feature extraction model and the second feature extraction model are optimized by using the parking environment image training set after data enhancement processing and the actual parking space information corresponding to the parking environment image to obtain the optimized first feature extraction model and the second feature extraction model.

9. A parking space detection device, characterized in that: The device comprises: An image capturing module, used to obtain the parking environment image captured by the vehicle-mounted image acquisition device; a feature map acquisition module, configured to input the parking environment image into a first feature extraction model to obtain an image feature map, wherein the first feature extraction model includes a first neural network capable of downsampling and extracting features from the parking environment image; A feature map conversion module, used to convert the image feature map into an environment feature map under a bird's-eye view by using a preset image pixel to vehicle coordinate system conversion relationship; a parking space feature extraction module, configured to input the environment feature map under the bird's-eye view into a second feature extraction model to obtain a parking space feature map under the bird's-eye view, wherein the second feature extraction model includes a second neural network capable of extracting target parking space features from the environment feature map; The parking space information detection module is used to detect and locate the parking space feature map under the bird's-eye view to obtain a parking space detection result.

10. A parking space detection system, characterized in that: The parking space detection system includes a controller, the controller includes a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the parking space detection method according to any one of claims 1 to 8 by executing the computer instructions.

11. A vehicle, characterized in that: The vehicle comprises the parking space detection system according to claim 10.

12. The vehicle according to claim 11, characterized in that The vehicle also includes a fisheye camera mounted on the vehicle.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the parking space detection method according to any one of claims 1 to 8.

Images