Parking space detection method, device, system, electronic equipment and program product

Abstract

The present application proposes a parking space detection method, device, system, electronic device and program product. The method determines feature data of multiple candidate parking space corner points based on image data of the environment in which the vehicle is located. The feature data of the candidate parking space corner points include the position features and image features of the candidate parking space corner points, and determines the parking space features corresponding to each candidate parking space corner point according to the feature data of each candidate parking space corner point, so as to predict the target detection information of the candidate parking space corresponding to each candidate parking space corner point according to the parking space features corresponding to each candidate parking space corner point, and the target detection information includes the first parking space detection information or the second parking space detection information. The first parking space detection information includes the required parking space information of the first parking stage, and the second parking space detection information includes the required parking space information of the second parking stage. The above-mentioned parking space detection scheme can effectively improve the parking space detection accuracy, thereby providing technical support for the automatic parking system to make correct parking decisions.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a parking space detection method, device, system, electronic equipment and program product. Background Art

[0002] With the rapid development of automobile technology, the vehicle occupancy rate has greatly increased, which has led to a rapid increase in the difficulty of parking. As a result, automatic parking technology has developed rapidly. In the process of automatic parking, the detection accuracy of the parking space is an important factor affecting the decision of automatic parking.

[0003] Currently, vision-based parking space detection methods are widely used in automatic parking technology. However, existing vision-based parking space detection methods have the problem of low detection accuracy. Summary of the invention

[0004] Based on the above requirements, the present application proposes a parking space detection method, device, system, electronic equipment and program product to solve the problem of low parking space detection accuracy in the prior art.

[0005] In order to achieve the above technical objectives, this application specifically proposes the following technical solutions: The first aspect of the present application provides a parking space detection method, comprising: Determine feature data of a plurality of candidate parking space corner points based on image data of the environment in which the vehicle is located; the feature data of the candidate parking space corner points includes position features and image features of the candidate parking space corner points; According to the feature data of each of the candidate parking space corner points, the parking space features corresponding to each of the candidate parking space corner points are determined respectively, and according to the parking space features corresponding to each of the candidate parking space corner points, the target detection information of the candidate parking space corresponding to each of the candidate parking space corner points is predicted; The target detection information includes first parking space detection information or second parking space detection information; the first parking space detection information includes required parking space information of a first parking stage, and the second parking space detection information includes required parking space information of a second parking stage.

[0006] The second aspect of the present application provides a parking space detection device, comprising: A first data processing module is used to determine feature data of a plurality of candidate parking space corner points based on image data of the environment in which the vehicle is located; the feature data of the candidate parking space corner points includes position features and image features of the candidate parking space corner points; A second data processing module is used to determine the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predict the target detection information of the candidate parking space corresponding to each of the candidate parking space corner points according to the parking space features corresponding to each of the candidate parking space corner points; The target detection information includes first parking space detection information or second parking space detection information; the first parking space detection information includes required parking space information of a first parking stage, and the second parking space detection information includes required parking space information of a second parking stage.

[0007] The third aspect of the present application provides a parking space detection system, comprising: Image acquisition device and data processing device; The data acquisition device is used to collect image data of the environment in which the vehicle is located; The data processing device is used to perform parking space detection based on the image data using the parking space detection method as described above.

[0008] In a fourth aspect, the present application proposes a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the parking space detection method as described above is implemented.

[0009] A fifth aspect of the present application provides an electronic device, comprising: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the above-mentioned parking space detection method by running the program in the memory.

[0010] In a sixth aspect, the present application proposes a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to execute the parking space detection method as described above.

[0011] The parking space detection method proposed in the present application determines feature data of multiple candidate parking space corner points based on image data of the environment in which the vehicle is located. The feature data of the candidate parking space corner points include position features and image features of the candidate parking space corner points. Since the parking space corner points are actual and more significant features in the parking space, the parking space detection can be performed more accurately based on the position features and image features of the parking space corner points. In addition, the parking space features corresponding to each candidate parking space corner point are determined respectively based on the feature data of each candidate parking space corner point, and the target detection information of the candidate parking space corresponding to each candidate parking space corner point is predicted respectively based on the parking space features corresponding to each candidate parking space corner point. The target detection information includes first parking space detection information or second parking space detection information. The first parking space detection information includes required parking space information in the first parking stage, and the second parking space detection information includes required parking space information in the second parking stage. Therefore, different parking space detection information can be provided in different parking stages, thereby further improving the parking space detection accuracy, and providing technical support for the automatic parking system to make correct parking decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0013] Figure 1 A schematic diagram of the structure of a parking space detection system provided in an embodiment of the present application.

[0014] Figure 2 A flowchart of a parking space detection method provided in an embodiment of the present application.

[0015] Figure 3 A schematic diagram of the structure of a parking space feature extraction model provided in an embodiment of the present application.

[0016] Figure 4 A schematic diagram of the structure of a decoder unit provided in an embodiment of the present application.

[0017] Figure 5 A schematic diagram of the structure of a parking space detection device provided in an embodiment of the present application.

[0018] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solution of the embodiment of the present application is applicable to application scenarios such as automatic parking of vehicles. The automatic parking system senses the surrounding environment of the vehicle through on-board sensors (such as cameras, laser radars, ultrasonic sensors, etc.), and detects parking spaces in combination with parking space detection algorithms to realize autonomous parking in and out of the vehicle. Automatic parking can not only reduce the burden on the driver and improve parking efficiency, but also effectively utilize limited parking space resources. By adopting the technical solution of the embodiment of the present application, parking space detection can be performed based on the feature data of the parking space corner points. Since the parking space corner points are actual and more significant features in the parking space, the parking space detection accuracy can be effectively improved; at the same time, through the technical solution of the embodiment of the present application, different parking space detection information can be provided in different parking stages, so that the parking space detection accuracy can be further improved, which provides technical support for the automatic parking system to make correct parking decisions, and can effectively solve the problem of difficult parking.

[0020] At present, the parking space detection process usually adopts the distance-based parking space detection method and the vision-based parking space detection method. Among them, the distance-based parking space detection method is mainly implemented by ultrasonic and laser radar sensors. Since this method cannot detect parking spaces with no vehicles parked around, it can generally only be used as a supplementary solution for parking space detection.

[0021] Vision-based parking space detection methods usually use on-board cameras to detect parking spaces around the vehicle in real time. This type of method is widely used due to its high accuracy and good adaptability. In the process of parking space detection, vision-based parking space detection methods usually use edge detection and Hough transform to identify parking space lines. They can only identify parking space information in simple environments, and the parking space detection results mainly depend on edge detection results. Therefore, the selection of hyperparameters such as prior values ​​and ROI (Region Of Interest) in the image processing process will directly affect the parking space detection accuracy, especially in scenarios where some key points of the parking space are blocked, light interference, aging and missing parking space lines, etc., the parking space detection accuracy is poor.

[0022] With the continuous development of deep learning technology, Faster R-CNN, YOLO and other deep learning-based object detection technologies are gradually applied to the field of parking space detection. Parking space detection methods based on deep learning usually predict the center point of the parking space, the length of the parking space, and the width of the parking space based on a rectangular box. However, since the center point of the parking space itself is a virtual point, it is difficult to accurately detect and locate it, especially when the rear corner point of the parking space is blurred or blocked. The detection accuracy of the center point will drop sharply, which will lead to a sharp drop in the accuracy of parking space detection. At the same time, in different parking stages of the vehicle, the parking space information contained in the collected image data is also different. Therefore, the existing methods perform parking space detection according to fixed modeling in different parking stages and cannot guarantee the accuracy of the parking space detection results.

[0023] Based on the above-mentioned technical status, the inventors of the present application have proposed a new parking space detection scheme after research. The parking space detection scheme can perform parking space detection based on the feature data of the parking space corner points. Since the parking space corner points are actual and relatively significant features in the parking space, the parking space detection accuracy can be effectively improved. At the same time, the parking space detection scheme can provide different parking space detection information in different parking stages, so as to further improve the parking space detection accuracy, provide technical support for the automatic parking system to make correct parking decisions, and thus effectively solve the problem of difficult parking.

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] Figure 1 An exemplary implementation environment of the technical solution proposed in the embodiment of the present application is shown, and the implementation environment is a parking space detection system. Figure 1 As shown, the system at least includes an image acquisition device 101 and a data processing device 102. The parking space detection system can be arranged on any vehicle, so as to provide the vehicle with a parking space detection function to assist the vehicle in automatic parking.

[0026] The image acquisition device 101 can be used to sense the driving environment of the vehicle and collect environmental perception data. The image acquisition device 101 can include one or more image acquisition devices 101 arranged at any position of the vehicle, for example, including but not limited to fisheye cameras, optical cameras, infrared cameras, etc. The image acquisition device 101 can detect and record data of the parking environment of the vehicle, for example, it can collect parking space information in the parking environment of the vehicle, and provide information such as distance, speed, color, texture, etc. of the parking environment.

[0027] One or more services or applications capable of executing parking space detection are running on the data processing device 102. As an optional implementation, the data processing device 102 may be a computer, a processing chip, a server, or other equipment with data processing capabilities.

[0028] The image acquisition device 101 and the data processing device 102 can be connected via a wired or wireless communication link. When the two are connected via a wired communication link, they can be connected via any type of signal line such as a coaxial signal line, a digital signal line, an optical cable signal line, a balanced signal line, etc. When the two are connected via a wireless communication link, they can be connected to each other via a network. The network can be any type of network. For example, the network can be a network that can be subdivided into multiple subnetworks. The network or the multiple subnetworks contained in the network can be a local area network (LAN), Ethernet, token ring, wide area network (WAN), Internet, virtual network, virtual private network (VPN), intranet, extranet, public switched telephone network (PSTN), infrared network, wireless network (such as Bluetooth, WIFI), mobile network (such as 4G, 5G), Internet of Things and / or any combination of these and / or other networks.

[0029] Furthermore, the above-mentioned parking space detection system may also include a memory for storing image data collected by the image acquisition device 101, or storing parking space detection results processed by the data processing device 102, or storing intermediate processing results of the data processing device 102, etc.

[0030] Combine the following Figure 1 The application scenario shown provides an exemplary introduction to the parking space detection solution proposed in the embodiment of the present application.

[0031] See also Figure 2 The embodiment of the present application first proposes a parking space detection method, which can be executed by a processor disposed on a vehicle. The method includes: S201. Determine feature data of a plurality of candidate parking corner points based on image data of an environment in which a vehicle is located; the feature data of the candidate parking corner points includes position features and image features of the candidate parking corner points.

[0032] Specifically, the environment in which the vehicle is located may include the environment in which the vehicle is located during the automatic parking process. The image data of the environment in which the vehicle is located may be determined based on the environmental image collected by the vehicle for parking space detection. For example, the image data of the environment in which the vehicle is located may include images collected by one or more image acquisition devices at the current moment, and may also include the splicing result of images collected by multiple image acquisition devices at the current moment, such as a bird's-eye view, etc.

[0033] For any image data of the environment in which the vehicle is located, image recognition can be performed on the image data to obtain a feature map corresponding to the image data, and feature data of the candidate parking corner points in the image data can be determined based on the feature map corresponding to the image data. For example, parking corner point recognition can be performed based on the feature map corresponding to the image data to obtain multiple candidate parking corner points, and feature data of each candidate parking corner point can be determined based on the position of each candidate parking corner point in the feature map.

[0034] The candidate parking corner points in the image data may include detection results of the parking corner points obtained by performing image recognition on the image data. For any candidate parking corner point, the feature data of the candidate parking corner point may include the position feature of the candidate parking corner point, and may also include the image feature of the candidate parking corner point. The position feature of the candidate parking corner point may represent the coordinates of the candidate parking corner point in the coordinate system where the vehicle is located. For example, the coordinates of the candidate parking corner point in the coordinate system where the vehicle is located may be encoded to obtain the position feature of the candidate parking corner point. The image feature of the candidate parking corner point may be used to represent the image feature of the image data corresponding to the candidate parking corner point at the candidate parking corner point.

[0035] S202, determining the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predicting the target detection information of the candidate parking space corresponding to each of the candidate parking space corner points according to the parking space features corresponding to each of the candidate parking space corner points; The target detection information includes first parking space detection information or second parking space detection information; the first parking space detection information includes required parking space information of a first parking stage, and the second parking space detection information includes required parking space information of a second parking stage.

[0036] Specifically, for any candidate parking space corner point, the parking space feature corresponding to the candidate parking space corner point can be determined based on the feature data of the candidate parking space corner point and the feature data of the target parking space corner point corresponding to the candidate parking space corner point. The target parking space corner point corresponding to the candidate parking space corner point may include a candidate parking space corner point whose distance from the candidate parking space corner point is less than a predetermined distance, may also include a candidate parking space corner point adjacent to the position of the candidate parking space corner point, and may also include all candidate parking space corner points, which may be specifically set according to actual needs.

[0037] The parking space features corresponding to the candidate parking space corner point may include features for characterizing the positions of the parking space corner points of the parking space where the candidate parking space corner point is located, features for characterizing the angle of the parking space where the candidate parking space corner point is located, features for characterizing the score of the parking space where the candidate parking space corner point is located, and features for characterizing the confidence of some or all of the parking space corner points of the parking space where the candidate parking space corner point is located. The specific settings can be made according to actual needs.

[0038] In implementation, the feature data of the candidate parking space corner point and the feature data of the target parking space corner point corresponding to the candidate parking space corner point may be input into a pre-trained parking space recognition model, so that the parking space feature corresponding to the candidate parking space corner point may be determined by the parking space recognition model according to the feature data of the candidate parking space corner point and the feature data of the target parking space corner point corresponding to the candidate parking space corner point. In addition, the feature data of all the candidate parking space corner points may be input into a pre-trained parking space recognition model, so that the parking space feature corresponding to each candidate parking space corner point may be automatically identified by the parking space recognition model according to the feature data of all the candidate parking space corner points, so that the parking space feature corresponding to each candidate parking space corner point may be quickly and accurately obtained.

[0039] For any candidate parking space corner point, the target detection information of the candidate parking space corresponding to the candidate parking space corner point can also be predicted according to the parking space features corresponding to the candidate parking space corner point. The candidate parking space corresponding to the candidate parking space corner point is the parking space where the candidate parking space corner point is located, which is determined according to the feature data of the candidate parking space corner point and the feature data of the target parking space corner point corresponding to the candidate parking space corner point.

[0040] The target detection information of the candidate parking space corresponding to the candidate parking space corner point may include the first parking space detection information or the second parking space detection information. The first parking space detection information may include the required parking space information of the first parking stage, and the second parking space detection information may include the required parking space information of the second parking stage. The parking stage of the vehicle may include the parking space search stage, the early parking stage, the mid-to-late parking stage, etc. Specifically, the parking stage of the vehicle may be divided into the first parking stage and the second parking stage according to the difference in the demand for parking space information. For example, the parking space search stage and the early parking stage may be divided into the first parking stage, and the mid-to-late parking stage may be divided into the second parking stage.

[0041] The required parking space information of the first parking stage may include the parking space information that the vehicle needs to refer to in the first parking stage, for example, it may include the position information of the parking space corner point on the parking space entrance line, that is, the position information of the front corner point of the parking space, and may also include the angle information of the parking space, for example, the angle of the two side lines of the parking space. The required parking space information of the second parking stage may include the parking space information that the vehicle needs to refer to in the second parking stage, for example, it may include the position information of the rear corner point of the parking space.

[0042] It is understandable that after obtaining the target detection information of the candidate parking spaces corresponding to the corner points of each candidate parking space, the vehicle can also determine the target parking space according to the target detection information of each candidate parking space, and realize automatic parking according to the target detection information of the target parking space. For example, the candidate parking space with the highest score can be used as the target parking space, and the candidate parking space closest to the vehicle can also be used as the target parking space. The target parking space can also be determined from the candidate parking spaces that can meet the required parking space information of the current parking stage according to the current parking stage of the vehicle and the target detection information of each candidate parking space. For example, when the current parking stage is the first parking stage, the target parking space can be determined from the candidate parking spaces whose target detection information is the first parking space detection information.

[0043] From the above introduction, it can be seen that the parking space detection method proposed in the embodiment of the present application can perform parking space detection based on the feature data of the parking space corner points. Since the parking space corner points are actual and significant features in the parking space, the parking space detection accuracy can be effectively improved. At the same time, the method of the embodiment of the present application can provide different parking space detection information at different parking stages, so that in the process of performing the corresponding parking space information detection, the degree of dependence on features other than the parking space detection information can be effectively reduced, and then when some corner points of the parking space are relatively blurred or blocked, the parking space detection information can still be accurately obtained, thereby achieving a further improvement in the parking space detection accuracy, thereby providing technical support for the automatic parking system to make correct parking decisions.

[0044] In some embodiments, the image data of the environment in which the vehicle is located includes image data acquired by a plurality of image acquisition devices; The feature data of the candidate parking space corner point also includes a coding feature of the image acquisition device corresponding to the candidate parking space corner point; wherein the dimensions of the coding feature, the position feature and the image feature are the same.

[0045] Specifically, multiple image acquisition devices can be respectively arranged in multiple different directions of the vehicle. For example, they can include image acquisition devices arranged on the front, rear, left and right sides of the vehicle, so that during the automatic parking process of the vehicle, image data of the vehicle's environment can be comprehensively and effectively collected.

[0046] The image data of the environment in which the vehicle is located may include image data collected by each of the multiple image acquisition devices. In implementation, for the image data collected by each image acquisition device at the current moment, feature data of the candidate parking space corner points in each image data may be determined respectively, thereby obtaining feature data of multiple candidate parking space corner points.

[0047] For any candidate parking space corner point, the feature data of the candidate parking space corner point may also include the coding feature of the image acquisition device corresponding to the candidate parking space corner point. The image acquisition device corresponding to the candidate parking space corner point is the image acquisition device that acquires the image data of the candidate parking space corner point among multiple image acquisition devices.

[0048] For any image acquisition device, the coding feature of the image acquisition device is used to uniquely identify the image acquisition device. For example, the identification information of the image acquisition device can be encoded to obtain the coding feature of the image acquisition device. The identification information of the image acquisition device may include the serial number of the image acquisition device, the identification code of the image acquisition device, the name of the image acquisition device, etc.

[0049] In implementation, for any candidate parking space corner point, the dimensions of the coding features of the image acquisition device corresponding to the candidate parking space corner point and the position features and image features of the candidate parking space corner point can be the same, so that in the process of determining the parking space features corresponding to each candidate parking space corner point according to the feature data of each candidate parking space corner point, the coding features, position features and image features corresponding to each candidate parking space corner point can be fused, and the parking space features corresponding to each candidate parking space corner point can be accurately and effectively determined. For example, the coding features, position features and image features corresponding to the candidate parking space corner points can be added to achieve feature fusion.

[0050] In the related art, the original images collected by multiple image acquisition devices are usually spliced ​​to obtain a bird's-eye view splicing image, and parking space detection is performed through the bird's-eye view splicing image to reduce the difficulty of parking space detection. For example, the original images collected by multiple image acquisition devices are subjected to IPM (Inverse Perspective Mapping) transformation to obtain a bird's-eye view splicing image. However, in the process of image splicing, a lot of original information in the collected images will be lost, reducing the accuracy of the bird's-eye view splicing image. At the same time, there is an assumption that the ground is horizontal in the IPM transformation process, which will cause transformation errors in uneven ground scenes, further reducing the accuracy of the bird's-eye view splicing image. Therefore, in the process of parking space detection through a bird's-eye view splicing image with poor accuracy, the accuracy of parking space detection cannot be guaranteed. The parking space detection method proposed in the embodiment of the present application can directly use the original images collected by each image acquisition device to perform parking space detection, thereby effectively avoiding the influence of splicing the original image data collected by each image acquisition device on the parking space detection accuracy, thereby achieving further improvement in the parking space detection accuracy.

[0051] During implementation, the specific method of obtaining the coding features of the image acquisition device corresponding to the corner point of the candidate parking space can be set according to actual needs. For example, the identification information of the image acquisition device corresponding to the corner point of the candidate parking space can be encoded through sine coding, cosine coding, machine learning network, etc. to obtain the coding features of the image acquisition device corresponding to the corner point of the candidate parking space.

[0052] Taking four image acquisition devices as an example, if the identification information of the image acquisition device is the number m of the image acquisition device, m∈[1,2,3,4], in the process of obtaining the coding feature of the image acquisition device, the number m of the image acquisition device can be converted into a one-hot (one-bit effective coding) vector, such as when m=1, the one-hot vector can be 1000, when m=2, the one-hot vector can be 0100, when m=3, the one-hot vector can be 0010, and when m=4, the one-hot vector can be 0001. Thus, the one-hot vector corresponding to the image acquisition device can be input into the target network model to obtain the coding feature of the image acquisition device. Taking the dimension of the coding feature as 256 dimensions as an example, the target network model can be a network model with 4 dimensions as input and 256 dimensions as output. For example, the target network model can include two fully connected layers, thereby enabling the image acquisition device corresponding to each candidate parking space corner point to be quickly and effectively encoded to obtain the corresponding coding feature.

[0053] In some embodiments, the step of determining the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predicting the target detection information of the candidate parking spaces corresponding to each of the candidate parking space corner points according to the parking space features corresponding to each of the candidate parking space corner points includes: The feature data of each of the candidate parking space corner points is input into a pre-trained parking space recognition model, and the parking space recognition model determines the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predicts the target detection information of the candidate parking spaces corresponding to each of the candidate parking space corner points based on the parking space features corresponding to each of the candidate parking space corner points.

[0054] Specifically, the parking space recognition model may be a deep learning model, for example, a convolutional neural network model, a Transformer model, etc. In implementation, the parking space recognition model may be trained in advance, so that during the parking process, the target detection information of the candidate parking space corresponding to each candidate parking space corner point may be quickly and accurately predicted by the pre-trained parking space recognition model.

[0055] During implementation, the feature data of each candidate parking space corner point can be input into a pre-trained parking space recognition model, so that the parking space features corresponding to each candidate parking space corner point can be determined by the parking space recognition model according to the feature data of each candidate parking space corner point, and the target detection information of the candidate parking spaces corresponding to each candidate parking space corner point can be predicted according to the parking space features corresponding to each candidate parking space corner point, thereby enabling the target detection information of the candidate parking spaces corresponding to each candidate parking space corner point to be quickly and accurately predicted.

[0056] In some embodiments, the parking space recognition model includes a parking space feature extraction model and a prediction model; Inputting the feature data of each of the candidate parking space corner points into a pre-trained parking space recognition model, the parking space recognition model determines the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predicting the target detection information of the candidate parking space corresponding to each of the candidate parking space corner points according to the parking space features corresponding to each of the candidate parking space corner points, including: Inputting the feature data of each of the candidate parking space corner points into the parking space feature extraction model, and the parking space feature extraction model determines the parking space feature corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points; The parking space features corresponding to the candidate parking space corner points are input into the prediction model, and the prediction model determines the target detection information of the candidate parking space corresponding to the candidate parking space corner points according to the parking space features corresponding to the candidate parking space corner points.

[0057] Specifically, the parking space recognition model may include a parking space feature extraction model and a prediction model. The input end of the parking space feature extraction model can be used to input feature data of each candidate parking space corner point, so that the parking space feature corresponding to each candidate parking space corner point can be determined by the parking space feature extraction model according to the feature data of each candidate parking space corner point.

[0058] The output end of the parking space feature extraction model can be connected to the input end of the prediction model to output the parking space features corresponding to each candidate parking space corner point to the prediction model, so as to predict the target detection information of the candidate parking space corresponding to each candidate parking space corner point through the prediction model.

[0059] In implementation, after determining the feature data of each candidate parking space corner point, the feature data of all candidate parking space corner points can be input into the parking space feature extraction model, thereby, the parking space feature corresponding to each candidate parking space corner point can be quickly and accurately obtained through the parking space feature extraction model.

[0060] At the same time, the parking space feature extraction model can output the parking space features corresponding to each candidate parking space corner point to the prediction model, so that the prediction model can determine the first parking space detection information or the second parking space detection information of each candidate parking space corner point according to the parking space features corresponding to each candidate parking space corner point.

[0061] For example, for any candidate parking space corner point, the prediction model can determine the parameter value of the target parameter according to the parking space feature corresponding to the candidate parking space corner point, and the target parameter can include the confidence of the parking space corner point on the parking space entrance line and / or the confidence of the parking space tail corner point, and select the first parking space detection information of the candidate parking space corresponding to the candidate parking space corner point to be predicted and output through the branch model corresponding to the first parking space detection information, or the second parking space detection information of the candidate parking space corresponding to the candidate parking space corner point to be predicted and output through the branch model corresponding to the second parking space detection information according to the parameter value of the target parameter. In addition, the prediction model can also simultaneously predict the first parking space detection information of the candidate parking space corresponding to the candidate parking space corner point through the branch model corresponding to the first parking space detection information, and predict and output the second parking space detection information of the candidate parking space corresponding to the candidate parking space corner point through the branch model corresponding to the second parking space detection information, and determine to output the first parking space detection information or the second parking space detection information according to the first parking space detection information and the second parking space detection information.

[0062] Therefore, through the embodiment scheme of the present application, the target detection information of the candidate parking space corresponding to each candidate parking space corner point can be determined quickly and accurately.

[0063] In some embodiments, the first parking space detection information includes parking space angle information and position information of a parking space corner point on a parking space entrance line; the second parking space detection information includes position information of a parking space rear corner point and a confidence level of the parking space rear corner point.

[0064] Specifically, the first parking space detection information may include parking space angle information, for example, the angles of two side lines of the corresponding candidate parking space. At the same time, the first parking space detection information may also include the position information of the parking space corner points on the parking space entrance line, so that in the first parking stage (such as the parking space search stage or the early stage of parking), the target parking space can be quickly and effectively determined and the corresponding parking strategy can be formulated according to the first parking space detection information. At the same time, in the process of determining the first parking space detection information, even if part of the parking space rear corner points of the corresponding parking space are blurred or blocked, the first parking space detection information can still be accurately determined.

[0065] The second parking space detection information may include the position information of the parking space tail corner point of the corresponding candidate parking space, so that in the second parking stage (e.g., the later stage of parking), the target parking space can be quickly and effectively determined and the corresponding parking strategy can be formulated according to the position information of the parking space tail corner point. At the same time, in the process of determining the second parking space detection information, when the parking space corner point on the entrance line of the corresponding parking space is blurred or blocked, the second parking space detection information can still be accurately determined. In addition, the second parking space detection information may also include the confidence of the parking space tail corner point of the corresponding candidate parking space, so that the prediction model can quickly and effectively determine and output the first parking space detection information or the second parking space detection information of the corresponding candidate parking space according to the confidence of the parking space tail corner point.

[0066] In some embodiments, the prediction model includes a first branch model, a second branch model and an output layer, and the input end of the output layer is connected to the output ends of the first branch model and the second branch model; The step of inputting the parking space features corresponding to the candidate parking space corner points into the prediction model, and determining the target detection information of the candidate parking space corresponding to the candidate parking space corner points by the prediction model according to the parking space features corresponding to the candidate parking space corner points, comprises: Inputting the parking space features corresponding to the candidate parking space corner points into the first branch model and the second branch model, the first branch model determining the first parking space detection information of the candidate parking space corresponding to the candidate parking space corner points according to the parking space features corresponding to the candidate parking space corner points, and the second branch model determining the second parking space detection information of the candidate parking space corresponding to the candidate parking space corner points according to the parking space features corresponding to the candidate parking space corner points; The output layer outputs the first parking space detection information or the second parking space detection information according to the confidence of the rear corner point of the parking space.

[0067] Specifically, the prediction model may include multiple branch models, and the multiple branch models are used to predict different parking space detection information. For example, the multiple branch models may include a first branch model and a second branch model, and the first branch model may be used to predict the first parking space detection information, and the second branch model may be used to predict the second parking space detection information; in addition, the multiple branch models may also include a third branch model, and the third branch model may be used to predict the parking space score of the candidate parking space corresponding to the corresponding candidate parking space corner point, and the parking space score may be used to characterize the credibility of the prediction result of the corresponding candidate parking space.

[0068] In addition, the prediction model may further include an output layer, the input end of the output layer may be connected to the output end of each branch model, and is used to determine the target detection information from the parking space detection information output by each branch model, and output the target detection information.

[0069] During implementation, for any candidate parking space corner point, the parking space features corresponding to the candidate parking space corner point can be respectively input into each branch model in the prediction model, thereby, each branch model can determine the corresponding parking space detection information according to the parking space features corresponding to the candidate parking space corner point, and output it to the output layer.

[0070] Among them, the first branch model can determine the first parking space detection information of the candidate parking space corresponding to the candidate parking space corner point according to the parking space features corresponding to the candidate parking space corner point, and the second branch model can determine the second parking space detection information of the candidate parking space corresponding to the candidate parking space corner point according to the parking space features corresponding to the candidate parking space corner point. In addition, the third branch model can also determine the parking space score of the candidate parking space corresponding to the candidate parking space corner point according to the parking space features corresponding to the candidate parking space corner point.

[0071] In the process of training the prediction model, the Hungarian matching algorithm can be used to distribute positive and negative samples so as to train the prediction model through positive and negative samples. For the parking space score prediction part and the parking space tail corner point confidence prediction part of the prediction model, the cross entropy loss function can be used for training. For the parking space tail corner point position prediction part, the parking space corner point position prediction part on the parking space entrance line and the parking space angle prediction part of the prediction model, the L1 loss function or the L2 loss function can be used for training to ensure the training efficiency of the prediction model and the reliability of the training results.

[0072] It can be understood that the parking space feature extraction model may include multiple output ports corresponding to multiple branch models, and each output port can be used to output different parking space features to the corresponding branch model in the prediction model, thereby, each branch model can predict the parking space detection information according to the corresponding parking space features. For example, the parking space features may include features for characterizing the parking space angle, features for characterizing the parking space corner point position on the parking space entrance line, features for characterizing the parking space tail corner point position, features for characterizing the parking space tail corner point confidence, and features for characterizing the parking space score, thereby, the features for characterizing the parking space angle and the features for characterizing the parking space corner point position on the parking space entrance line can be output to the first branch model, the features for characterizing the parking space tail corner point position and the features for characterizing the parking space tail corner point confidence can be output to the second branch model, and the features for characterizing the parking space score can be output to the third branch model.

[0073] After receiving the parking space detection information output by each branch model, the output layer can output the parking space score of the corresponding candidate parking space, and at the same time, can determine whether to output the first parking space detection information or the second parking space detection information of the corresponding candidate parking space according to the confidence of the parking space tail corner point in the second parking space detection information. For example, when the confidence of the parking space tail corner point is greater than or equal to a predetermined threshold, the second parking space detection information can be output, and when the confidence of the parking space tail corner point is less than the predetermined threshold, the first parking space detection information can be output.

[0074] Therefore, through the embodiment of the present application, the first parking space detection information or the second parking space detection information that meets the parking demand can be output quickly and accurately.

[0075] Optionally, the parking space feature extraction model may adopt a Transformer decoder model. The Transformer decoder may establish a global connection among the candidate parking space corner points through a self-attention mechanism, and determine the parking space features corresponding to the candidate parking space corner points based on the global connection. At the same time, the Transformer decoder may also determine the candidate parking space corner points in the same parking space through a self-attention mechanism, and assign different parking space scores to the candidate parking space corresponding to each candidate parking space corner point based on the feature data of the identified candidate parking space corner points in the same parking space. For example, the positioning credibility of the candidate parking space corner point may be determined based on the feature data of the candidate parking space corner point. The candidate parking space corner point with higher positioning credibility has a higher corresponding parking space score.

[0076] Among them, Figure 3 As shown, the parking space feature extraction model may include a plurality of decoder units connected in sequence, and each decoder unit may include a multi-head attention layer and a feedforward network layer connected in sequence, thereby effectively improving the parking space feature recognition accuracy of the parking space feature extraction model.

[0077] For any decoder unit, the multi-head attention layer and the feed-forward network layer in the decoder unit can both use residual connections, for example, Figure 4 As shown, a first residual block can be set between the multi-head attention layer and the feedforward network layer, and a second residual block can be set between the feedforward network layer and the output end of the decoder unit. The first residual block can be used to add the input and output of the multi-head attention layer and perform normalization processing, and output the normalization processing result to the feedforward network layer. The second residual block can be used to add the input and output of the feedforward network layer and perform normalization processing, and output the normalization processing result to the multi-head attention layer in the next decoder unit, or, as the final parking space feature output to the prediction model. In this way, the parking space feature recognition accuracy of the parking space feature extraction model can be further improved.

[0078] In some embodiments, determining feature data of a plurality of candidate parking space corner points based on image data of the environment in which the vehicle is located includes: Obtain a feature map corresponding to the image data, and obtain a probability corresponding to each pixel in the feature map, where the probability corresponding to the pixel is used to represent the probability that the position corresponding to the pixel is a corner point of a parking space; Determine the candidate parking space corner points corresponding to the image data based on the probabilities corresponding to the pixels in the feature map; Based on the position information and image features of the pixels corresponding to the candidate parking space corner points in the feature map, the position features and image features of the candidate parking space corner points are determined respectively.

[0079] Specifically, for any image data, a feature map corresponding to the image data can be obtained, and the feature map corresponding to the image data can include features of the image data at different levels, for example, edge features, texture features, color features, etc. of the image data at different resolutions. In implementation, the feature map corresponding to the image data can be extracted through an image recognition model.

[0080] For any image data, the feature map corresponding to the image data may include multiple pixels. For example, for a feature map of size 20*20, the feature map may include 20*20 pixels. For any pixel in the feature map, the probability corresponding to the pixel may be used to characterize the probability that the position corresponding to the pixel is a corner point of a parking space.

[0081] Among them, the probability that the position corresponding to the pixel is a parking space corner point can be identified based on the image feature of the feature map at the pixel. For example, the feature map corresponding to each image data can be input into a pre-trained corner point detection model, so that the corner point detection model can obtain a parking space corner point heatmap according to the feature map corresponding to each image data. The parking space corner point heatmap can represent the probability of different elements being parking space corner points by color intensity. The training of the corner point detection model can be supervised by the cross entropy loss function, so as to effectively ensure the reliability of the output result of the corner point detection model.

[0082] In implementation, the candidate parking space corner points corresponding to each image data may be determined based on the probability corresponding to each pixel in the feature map. For example, the probabilities corresponding to each pixel in the feature map of all image data may be sorted, and a target number of pixels may be selected as candidate parking space corner points in descending order of probability, such as the target number may be 128; in addition, pixels whose corresponding probabilities are greater than a predetermined probability may be selected as candidate parking space corner points.

[0083] For any candidate parking space corner point, the position feature of the candidate parking space corner point can be determined based on the position information of the pixel corresponding to the candidate parking space corner point in the feature map. The position information of the pixel corresponding to the candidate parking space corner point can be the position coordinates of the pixel corresponding to the candidate parking space corner point. The position feature of the candidate parking space corner point can be obtained by encoding the position information of the pixel corresponding to the candidate parking space corner point. The specific encoding method can be set according to actual needs. For example, the position information of the pixel corresponding to the candidate parking space corner point can be encoded by cosine encoding, sine encoding, machine learning network, etc.

[0084] Optionally, the position information of the pixel corresponding to the corner point of the candidate parking space may be encoded by using both sine encoding and cosine encoding. For example, taking the dimension of the position feature as 256 dimensions, if the position information of the pixel corresponding to the corner point of the candidate parking space is (pos x ,pos y ), pos x and pos y are the coordinates of the pixels corresponding to the candidate parking space corner points in the x-direction and the y-direction respectively. Then, the 256-dimensional position feature of the candidate parking space corner points can be determined as shown in formula (1): (1) In the formula, pos x The corresponding 2i-th dimension position feature, pos x The corresponding (2i+1)th dimension position feature, posy The corresponding 2i-th dimension position feature, pos y The corresponding (2i+1)th dimension position feature, i∈[0,1,…,63]. Thus, we can get pos x Corresponding 128-dimensional features and pos y The corresponding 128-dimensional features, pos x Corresponding 128-dimensional features and pos y The corresponding 128-dimensional features can be concatenated to obtain the 256-dimensional position features of the candidate parking space corner points.

[0085] In addition, for any candidate parking space corner point, the image features of the candidate parking space corner point can be determined based on the image features of the pixels corresponding to the candidate parking space corner point in the feature map. For example, the image features of the pixels corresponding to the candidate parking space corner point can be directly used as the image features of the candidate parking space corner point, and the image features of the pixels corresponding to the candidate parking space corner point can be further feature extracted to obtain the image features of the candidate parking space corner point.

[0086] Therefore, through the method of the embodiment of the present application, the candidate parking space corner points in each image data and the position features and image features of each candidate parking space corner point can be quickly and effectively determined.

[0087] In some embodiments, obtaining a feature map corresponding to the image data includes: The image data is input into a pre-trained image recognition model, and the image recognition model performs feature extraction on the image data to obtain a feature map corresponding to the image data.

[0088] Specifically, the image recognition model may be a deep learning model, for example, a convolutional neural network model, a graph neural network model, etc. In implementation, the image recognition model may be trained in advance, so that during the parking process, the pre-trained image recognition model may be used to quickly and accurately extract features from each image data to obtain a feature map corresponding to each image data.

[0089] During implementation, for any image data, the image data can be input into a pre-trained image recognition model, and the image recognition model can perform feature extraction on the image data to obtain a feature map corresponding to the image data, so that the feature map corresponding to each image data can be obtained quickly and accurately, and then each candidate parking space corner point and the image features of each candidate parking space corner point can be accurately determined based on each feature map.

[0090] In some embodiments, the image recognition model includes a fusion layer and a plurality of branch networks, the plurality of branch networks correspond one-to-one to a plurality of predetermined downsampling resolutions, and the input end of the fusion layer is connected to the output end of each branch network; The step of inputting the image data into a pre-trained image recognition model, and extracting features from the image data by the image recognition model to obtain a feature map corresponding to the image data includes: Inputting the sampling graphs of the image data at each of the predetermined downsampling resolutions into the corresponding branch networks respectively, and performing feature extraction on the sampling graphs at the corresponding predetermined downsampling resolutions by each of the branch networks respectively, to obtain sub-feature graphs of the image data at each of the predetermined downsampling resolutions; The resampled feature maps of each of the sub-feature maps at the target resolution are input into the fusion layer, and the fusion layer fuses the resampled feature maps to obtain the feature map corresponding to the image data.

[0091] Specifically, the image recognition model may include a fusion layer and multiple branch networks, and the multiple branch networks may correspond one-to-one to multiple predetermined downsampling resolutions, that is, each branch network may be used to extract features from a sampling map of image data at a corresponding predetermined downsampling resolution to obtain a sub-feature map of the image data at the corresponding predetermined downsampling resolution. The fusion layer may be used to perform fusion processing on the sub-feature maps of the image data at each predetermined downsampling resolution to obtain a feature map corresponding to the image data. The predetermined downsampling resolution may be configured according to actual needs, for example, the predetermined downsampling resolution may include 1 / 8, 1 / 16, and 1 / 32.

[0092] During implementation, for any image data, the sampling graphs of the image data at each predetermined downsampling resolution can be input into the corresponding branch networks respectively, and each branch network performs feature extraction on the sampling graphs at the corresponding predetermined downsampling resolutions to obtain sub-feature graphs of each image data at each predetermined downsampling resolution, thereby obtaining sub-feature graphs of each image data at different levels.

[0093] Among them, the image data can be downsampled based on each predetermined downsampling resolution to obtain a sampling image of the image data at each predetermined downsampling resolution; a first feature extraction network can also be set in the image recognition model, and the first feature extraction network can include multiple output ports corresponding to multiple branch networks. The image data can be input into the first feature extraction network, and the resolution of the image data can be gradually reduced through the convolution layer or pooling layer in the first feature extraction network, and the sampling images at the corresponding predetermined downsampling resolutions can be output to the branch network through each output port.

[0094] In addition, the sub-feature map of the image data at each predetermined downsampling resolution may be resampled based on the target resolution to obtain a resampled feature map of the image data at the target resolution. For example, a second feature extraction network may be provided between each branch network and the fusion layer, and the second feature extraction network may resample the sub-feature map output by the branch network to obtain a resampled feature map at the target resolution, and output the resampled feature map at the target resolution to the fusion layer.

[0095] It can be understood that the target resolution can be the same as one of the predetermined downsampling resolutions. For example, the target resolution can be the same as the predetermined downsampling resolution with the highest resolution among the predetermined downsampling resolutions. For example, when the predetermined downsampling resolutions include 1 / 8, 1 / 16 and 1 / 32, the target resolution can be 1 / 8. At this time, the branch network corresponding to the predetermined downsampling resolution of 1 / 8 can be directly connected to the fusion layer, that is, the sub-feature map output by the branch network corresponding to the predetermined downsampling resolution of 1 / 8 can be used as the corresponding resampled feature map and directly input into the fusion layer.

[0096] Among them, the resampled feature maps of each sub-feature map at the target resolution can be input into the fusion layer, and the fusion layer fuses the resampled feature maps to obtain the feature map corresponding to the image data.

[0097] Therefore, through the method of the embodiment of the present application, it is possible to ensure that the feature map corresponding to each image data includes features at different levels, and then based on the feature map corresponding to each image data, the corner points of the candidate parking spaces can be accurately identified.

[0098] Corresponding to the above-mentioned parking space detection method, the present application embodiment also provides a parking space detection device, see Figure 5 As shown, the device comprises: The first data processing module 501 is used to determine feature data of a plurality of candidate parking space corner points based on the image data of the environment in which the vehicle is located; the feature data of the candidate parking space corner points includes position features and image features of the candidate parking space corner points; The second data processing module 502 is used to determine the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predict the target detection information of the candidate parking space corresponding to each of the candidate parking space corner points according to the parking space features corresponding to each of the candidate parking space corner points; The target detection information includes first parking space detection information or second parking space detection information; the first parking space detection information includes required parking space information of a first parking stage, and the second parking space detection information includes required parking space information of a second parking stage.

[0099] In a possible implementation manner, the image data of the environment in which the vehicle is located includes image data acquired by a plurality of image acquisition devices; The feature data of the candidate parking space corner point also includes a coding feature of the image acquisition device corresponding to the candidate parking space corner point; wherein the dimensions of the coding feature, the position feature and the image feature are the same.

[0100] In a possible implementation manner, the second data processing module 502 is specifically configured to: The feature data of each of the candidate parking space corner points is input into a pre-trained parking space recognition model, and the parking space recognition model determines the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predicts the target detection information of the candidate parking spaces corresponding to each of the candidate parking space corner points based on the parking space features corresponding to each of the candidate parking space corner points.

[0101] In a possible implementation manner, the parking space recognition model includes a parking space feature extraction model and a prediction model; the second data processing module 502 is specifically used to: Inputting the feature data of each of the candidate parking space corner points into the parking space feature extraction model, and the parking space feature extraction model determines the parking space feature corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points; The parking space features corresponding to the candidate parking space corner points are input into the prediction model, and the prediction model determines the target detection information of the candidate parking space corresponding to the candidate parking space corner points according to the parking space features corresponding to the candidate parking space corner points.

[0102] In a possible implementation manner, the first parking space detection information includes parking space angle information and position information of a parking space corner point on a parking space entrance line; the second parking space detection information includes position information of a parking space rear corner point and a confidence level of the parking space rear corner point.

[0103] In a possible implementation, the prediction model includes a first branch model, a second branch model and an output layer, the input end of the output layer is connected to the output ends of the first branch model and the second branch model; the second data processing module 502 is specifically used to: Inputting the parking space features corresponding to the candidate parking space corner points into the first branch model and the second branch model, the first branch model determining the first parking space detection information of the candidate parking space corresponding to the candidate parking space corner points according to the parking space features corresponding to the candidate parking space corner points, and the second branch model determining the second parking space detection information of the candidate parking space corresponding to the candidate parking space corner points according to the parking space features corresponding to the candidate parking space corner points; The output layer outputs the first parking space detection information or the second parking space detection information according to the confidence of the rear corner point of the parking space.

[0104] In a possible implementation manner, the first data processing module 501 is specifically configured to: Obtain a feature map corresponding to the image data, and obtain a probability corresponding to each pixel in the feature map, where the probability corresponding to the pixel is used to represent the probability that the position corresponding to the pixel is a corner point of a parking space; Determine the candidate parking space corner points corresponding to the image data based on the probabilities corresponding to the pixels in the feature map; Based on the position information and image features of the pixels corresponding to the candidate parking space corner points in the feature map, the position features and image features of the candidate parking space corner points are determined respectively.

[0105] In a possible implementation manner, the first data processing module 501 is specifically configured to: The image data is input into a pre-trained image recognition model, and the image recognition model performs feature extraction on the image data to obtain a feature map corresponding to the image data.

[0106] In a possible implementation, the image recognition model includes a fusion layer and multiple branch networks, the multiple branch networks correspond one-to-one to multiple predetermined downsampling resolutions, and the input end of the fusion layer is connected to the output end of each branch network; the first data processing module 501 is specifically used for: Inputting the sampling graphs of the image data at each of the predetermined downsampling resolutions into the corresponding branch networks respectively, and performing feature extraction on the sampling graphs at the corresponding predetermined downsampling resolutions by each of the branch networks respectively, to obtain sub-feature graphs of the image data at each of the predetermined downsampling resolutions; The resampled feature maps of each of the sub-feature maps at the target resolution are input into the fusion layer, and the fusion layer fuses the resampled feature maps to obtain the feature map corresponding to the image data.

[0107] The parking space detection device provided in this embodiment belongs to the same application concept as the parking space detection method provided in the above embodiments of this application, and can execute the parking space detection method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the parking space detection method. For technical details not fully described in this embodiment, please refer to the specific processing content of the parking space detection method provided in the above embodiments of this application, and will not be repeated here.

[0108] The functions implemented by the above modules can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0109] It should be understood that each module in the above device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the unit in the device can be implemented in the form of a hardware circuit, and the functions of some or all units can be realized by designing the hardware circuit. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all of the above units. All units of the above device can be implemented in the form of a processor calling software, or in the form of a hardware circuit, or in part by a processor calling software, and the remaining part is implemented in the form of a hardware circuit.

[0110] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0111] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0112] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0113] The embodiment of the present application also provides a parking space detection system, which includes at least an image acquisition device and a data processing device; The data acquisition device is used to collect image data of the environment in which the vehicle is located; The data processing device is used to perform parking space detection based on the image data using the parking space detection method described in any one of the above embodiments.

[0114] The above-mentioned parking space detection system can be installed on any vehicle to assist the vehicle in performing parking space detection, thereby assisting the vehicle in achieving automatic parking.

[0115] Another embodiment of the present application also provides an electronic device, see Figure 6 As shown, the electronic device includes: Memory 200 and processor 210; The memory 200 is connected to the processor 210 and is used to store programs; The processor 210 is used to implement the parking space detection method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0116] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0117] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus. A bus may include a pathway that transfers information between components of a computer system.

[0118] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0119] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0120] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0121] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0122] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0123] The communication interface 220 may include any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0124] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any parking space detection method provided in the above embodiments of the present application.

[0125] The method in the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions, and when the computer programs or instructions are loaded and executed on a computer, the process or function described in the present application is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM or other programmable device.

[0126] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0127] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0128] In addition, the embodiment of the present application may also be a storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the parking space detection method described in any of the above embodiments of this specification, and specifically the following steps may be implemented: S201. Determine feature data of a plurality of candidate parking corner points based on image data of an environment in which a vehicle is located; the feature data of the candidate parking corner points includes position features and image features of the candidate parking corner points.

[0129] S202, determining the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predicting the target detection information of the candidate parking space corresponding to each of the candidate parking space corner points according to the parking space features corresponding to each of the candidate parking space corner points; The target detection information includes first parking space detection information or second parking space detection information; the first parking space detection information includes required parking space information of a first parking stage, and the second parking space detection information includes required parking space information of a second parking stage.

[0130] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0131] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0132] The steps in the methods of the embodiments of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in the embodiments can be replaced or combined. The modules and submodules in the devices and terminals in the embodiments of the present application can be combined, divided and deleted according to actual needs.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0134] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0136] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0137] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0138] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A parking space detection method, characterized in that: include: Determine feature data of a plurality of candidate parking space corner points based on image data of the environment in which the vehicle is located; the feature data of the candidate parking space corner points includes position features and image features of the candidate parking space corner points; According to the feature data of each of the candidate parking space corner points, the parking space features corresponding to each of the candidate parking space corner points are determined respectively, and according to the parking space features corresponding to each of the candidate parking space corner points, the target detection information of the candidate parking space corresponding to each of the candidate parking space corner points is predicted; The target detection information includes first parking space detection information or second parking space detection information; the first parking space detection information includes required parking space information of a first parking stage, and the second parking space detection information includes required parking space information of a second parking stage.

2. The method according to claim 1, characterized in that The image data of the environment in which the vehicle is located includes image data collected by multiple image collection devices; The feature data of the candidate parking space corner point also includes a coding feature of the image acquisition device corresponding to the candidate parking space corner point; wherein the dimensions of the coding feature, the position feature and the image feature are the same.

3. The method according to claim 1, characterized in that The step of determining the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predicting the target detection information of the candidate parking spaces corresponding to each of the candidate parking space corner points according to the parking space features corresponding to each of the candidate parking space corner points, includes: The feature data of each of the candidate parking space corner points is input into a pre-trained parking space recognition model, and the parking space recognition model determines the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predicts the target detection information of the candidate parking spaces corresponding to each of the candidate parking space corner points based on the parking space features corresponding to each of the candidate parking space corner points.

4. The method according to claim 3, characterized in that The parking space recognition model includes a parking space feature extraction model and a prediction model; Inputting the feature data of each of the candidate parking space corner points into a pre-trained parking space recognition model, the parking space recognition model determines the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predicting the target detection information of the candidate parking space corresponding to each of the candidate parking space corner points according to the parking space features corresponding to each of the candidate parking space corner points, including: Inputting the feature data of each of the candidate parking space corner points into the parking space feature extraction model, and the parking space feature extraction model determines the parking space feature corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points; The parking space features corresponding to the candidate parking space corner points are input into the prediction model, and the prediction model determines the target detection information of the candidate parking space corresponding to the candidate parking space corner points according to the parking space features corresponding to the candidate parking space corner points.

5. The method according to claim 4, characterized in that The first parking space detection information includes parking space angle information and position information of a parking space corner point on a parking space entrance line; the second parking space detection information includes position information of a parking space rear corner point and a confidence level of the parking space rear corner point.

6. The method according to claim 5, characterized in that The prediction model includes a first branch model, a second branch model and an output layer, and the input end of the output layer is connected to the output ends of the first branch model and the second branch model; The step of inputting the parking space features corresponding to the candidate parking space corner points into the prediction model, and determining the target detection information of the candidate parking space corresponding to the candidate parking space corner points by the prediction model according to the parking space features corresponding to the candidate parking space corner points, comprises: Inputting the parking space features corresponding to the candidate parking space corner points into the first branch model and the second branch model, the first branch model determining the first parking space detection information of the candidate parking space corresponding to the candidate parking space corner points according to the parking space features corresponding to the candidate parking space corner points, and the second branch model determining the second parking space detection information of the candidate parking space corresponding to the candidate parking space corner points according to the parking space features corresponding to the candidate parking space corner points; The output layer outputs the first parking space detection information or the second parking space detection information according to the confidence of the rear corner point of the parking space.

7. The method according to any one of claims 1 to 6, characterized in that: The step of determining feature data of a plurality of candidate parking space corner points based on the image data of the environment in which the vehicle is located includes: Obtain a feature map corresponding to the image data, and obtain a probability corresponding to each pixel in the feature map, where the probability corresponding to the pixel is used to represent the probability that the position corresponding to the pixel is a corner point of a parking space; Determine the candidate parking space corner points corresponding to the image data based on the probabilities corresponding to the pixels in the feature map; Based on the position information and image features of the pixels corresponding to the candidate parking space corner points in the feature map, the position features and image features of the candidate parking space corner points are determined respectively.

8. The method according to claim 7, characterized in that The obtaining of a feature map corresponding to the image data includes: The image data is input into a pre-trained image recognition model, and the image recognition model performs feature extraction on the image data to obtain a feature map corresponding to the image data.

9. The method according to claim 8, characterized in that The image recognition model includes a fusion layer and a plurality of branch networks, the plurality of branch networks correspond one-to-one to a plurality of predetermined downsampling resolutions, and the input end of the fusion layer is connected to the output end of each branch network; The step of inputting the image data into a pre-trained image recognition model, and extracting features from the image data by the image recognition model to obtain a feature map corresponding to the image data includes: Inputting the sampling graphs of the image data at each of the predetermined downsampling resolutions into the corresponding branch networks respectively, and performing feature extraction on the sampling graphs at the corresponding predetermined downsampling resolutions by each of the branch networks respectively, to obtain sub-feature graphs of the image data at each of the predetermined downsampling resolutions; The resampled feature maps of each of the sub-feature maps at the target resolution are input into the fusion layer, and the fusion layer fuses the resampled feature maps to obtain the feature map corresponding to the image data.

10. A parking space detection device, characterized in that: include: A first data processing module is used to determine feature data of a plurality of candidate parking space corner points based on image data of the environment in which the vehicle is located; the feature data of the candidate parking space corner points includes position features and image features of the candidate parking space corner points; A second data processing module is used to determine the parking space features corresponding to each of the candidate parking space corner points according to the feature data of each of the candidate parking space corner points, and predict the target detection information of the candidate parking space corresponding to each of the candidate parking space corner points according to the parking space features corresponding to each of the candidate parking space corner points; The target detection information includes first parking space detection information or second parking space detection information; the first parking space detection information includes required parking space information of a first parking stage, and the second parking space detection information includes required parking space information of a second parking stage.

11. A parking space detection system, characterized in that: include: Image acquisition device and data processing device; The data acquisition device is used to collect image data of the environment in which the vehicle is located; The data processing device is used to perform parking space detection based on the image data using the parking space detection method as described in any one of claims 1 to 9.

12. An electronic device, characterized in that: include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the parking space detection method according to any one of claims 1 to 9 by running the program in the memory.

13. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to execute the parking space detection method according to any one of claims 1 to 9.

Images