A parking space detection method and device, electronic equipment and storage medium

Abstract

The application discloses a parking space detection method and device, electronic equipment and storage medium, and relates to the field of intelligent parking. The method comprises the following steps: performing feature extraction on a target feature image of a to-be-detected parking space image; performing parking space entrance information extraction based on the target feature image to obtain target parking space entrance information; the target parking space entrance information comprises a parking space entrance key point heat map, a parking space entrance center point heat map, parking space entrance direction information and position offset information; determining a target parking space entrance key point corresponding to each parking space entrance according to the parking space entrance key point heat map, the parking space entrance center point heat map and the position offset information; and determining a target parking space based on the target parking space entrance key point and the corresponding direction information. The application determines and corrects the parking space entrance key point by using the parking space entrance center point and the position offset information, and adds the parking space entrance direction information, so that the parking space detection result is more accurate.

Description

Technical Field

[0001] This invention relates to the field of intelligent parking, and in particular to a parking space detection method, device, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence, automatic parking technology has made significant progress, making it possible for vehicles to automatically park in various complex environments. Automatic parking refers to parking without human intervention; the car uses perception and control algorithms to park itself automatically. Users simply need to press a button in the car or via a mobile app, and the vehicle will automatically find an available parking space and park. Therefore, this technology has gained significant attention from major manufacturers, and they are continuously developing new technologies and algorithms to improve the accuracy of automatic parking. Parking space detection algorithms, as a prerequisite for automatic parking, are particularly important.

[0003] There are several common parking space detection methods, such as object detection, semantic segmentation, Transformer, or combinations of these methods. However, existing parking space detection methods are difficult to accurately detect parking spaces when the image information is incomplete, and they also involve large computational loads and are difficult to implement in engineering. Summary of the Invention

[0004] To address the problems of existing technologies, embodiments of the present invention provide a parking space detection method, apparatus, electronic device, and storage medium, the technical solution of which is as follows:

[0005] Firstly, a parking space detection method is provided, the method comprising:

[0006] A parking space image to be detected is acquired, and a target feature image is obtained by extracting features from the parking space image to be detected; the parking space image to be detected includes the entrance to at least one parking space;

[0007] Based on the target feature image, parking space entrance information is predicted to obtain target parking space entrance information. The target parking space entrance information includes a heatmap of key points at the parking space entrance, a heatmap of the center point of the parking space entrance, parking space entrance direction information, and position offset information. The heatmap of key points at the parking space entrance represents the confidence level that each pixel in the image of the parking space to be detected is a key point at the parking space entrance; the heatmap of the center point of the parking space entrance represents the confidence level that each pixel in the image of the parking space to be detected is the center point of the parking space entrance; the parking space entrance direction information represents the direction information corresponding to each pixel in the image of the parking space to be detected; and the position offset information represents the position offset amount corresponding to each pixel in the image of the parking space to be detected.

[0008] Based on the heat map of the key points of the parking space entrance, the heat map of the center point of the parking space entrance, and the position offset information, determine the target parking space entrance key points corresponding to each parking space entrance;

[0009] Determine the directional information corresponding to the key points of the target parking space entrance for each parking space entrance in the parking space entrance directional information;

[0010] The target parking space is determined based on the key point of the target parking space entrance corresponding to each parking space entrance and the direction information corresponding to the key point of the target parking space entrance.

[0011] Optionally, the step of predicting parking space entrance information based on the target feature image to obtain target parking space entrance information includes:

[0012] The target feature image is input into the first branch network of the parking space entrance information prediction network. The first branch network predicts the key points of the parking space entrance based on the target feature image to obtain the heat map of the key points of the parking space entrance.

[0013] The target feature image is input into the second branch network of the parking space entrance information prediction network. The second branch network predicts the center point of the parking space entrance based on the target feature image to obtain the heat map of the center point of the parking space entrance.

[0014] The target feature image is input into the third branch network of the parking space entrance information prediction network, and the third branch network predicts the parking space entrance direction information based on the target feature image to obtain the parking space entrance direction information.

[0015] The target feature image is input into the fourth branch network of the parking space entrance information prediction network, and the fourth branch network predicts the position offset information based on the target feature image to obtain the position offset information.

[0016] Optionally, determining the target parking space entrance key point corresponding to each parking space entrance based on the heat map of the parking space entrance key points, the heat map of the parking space entrance center point, and the position offset information includes:

[0017] Based on the pixels in the heat map of the parking space entrance center point that have a confidence level exceeding a preset confidence threshold, at least one parking space entrance center point is obtained; each parking space entrance center point corresponds to one parking space entrance.

[0018] Determine the position offset of the center point of each parking space entrance in the position offset information;

[0019] Based on the center point of each parking space entrance and the position offset corresponding to the center point of the parking space entrance, determine the first parking space entrance key point associated with each of the parking space entrance center points;

[0020] From the heat map of key points at parking space entrances, within a preset range centered on the first key point at parking space entrance, determine the target key point at each parking space entrance.

[0021] Optionally, the target parking space entrance information further includes position compensation information, which represents the position compensation amount of each pixel in the image of the parking space to be detected. The step of extracting the parking space entrance information based on the target feature image to obtain the target parking space entrance information further includes:

[0022] The target feature image is input into the fifth branch network of the parking space entrance information prediction network, and the fifth branch network predicts the location compensation information based on the target feature image to obtain the location compensation information.

[0023] The step of determining the target parking space entrance key point corresponding to each parking space entrance within a preset range centered on the first parking space entrance key point from the heat map of the parking space entrance key points includes:

[0024] Based on the pixel with the highest confidence level within a preset range centered on the first parking space entrance key point in the heat map of the parking space entrance key points, the second parking space entrance key point corresponding to each parking space entrance is obtained.

[0025] Determine the location compensation amount corresponding to the key point of the second parking space entrance for each parking space entrance in the location compensation information;

[0026] Based on the key point of the second parking space entrance corresponding to each parking space entrance and the position compensation amount corresponding to the key point of the second parking space entrance, the key point of the target parking space entrance corresponding to each parking space entrance is determined.

[0027] Optionally, determining the target parking space based on the key point of the target parking space entrance and the direction information corresponding to the key point of the target parking space entrance includes:

[0028] Based on the direction information corresponding to the key point of the target parking space entrance, an angle value is determined; the angle value represents the angle between the parking space entrance direction line connected to the key point of the target parking space entrance and the preset baseline in the clockwise direction;

[0029] The parking space entrance direction line is determined based on the angle value and the preset baseline.

[0030] The target parking space is determined based on the direction line of the parking space entrance and the key points of the target parking space entrance.

[0031] Optionally, the step of extracting features from the image of the parking space to be detected to obtain the target feature image includes:

[0032] The image of the parking space to be detected is downsampled a preset number of times to obtain an initial feature image sequence; the initial feature image sequence includes multiple initial feature images of different resolutions;

[0033] The target feature image is obtained by fusing features from the multiple initial feature images of different resolutions in a preset order.

[0034] Optionally, the step of fusing features from the plurality of initial feature images of different resolutions in a preset order to obtain the target feature image includes:

[0035] The initial feature image with the lowest resolution in the initial feature image sequence is taken as the current image, and the current image is upsampled.

[0036] According to a preset order, the adjacent initial feature images of the current image in the initial feature image sequence are taken as the first image, and the first image is subjected to convolution processing of a preset size;

[0037] The upsampled current image and the convolutionally processed first image are fused to obtain the fused feature image.

[0038] The current image is updated based on the fused feature image until the initial feature image with the highest resolution in the initial feature image sequence is taken as the first image, and the resulting fused feature image is taken as the target feature image.

[0039] Optionally, the parking space detection method further includes a training process for a prediction network that provides the parking space entrance information, the training process including:

[0040] Construct an initial parking space entrance information extraction network to be trained;

[0041] A sample target feature image is obtained, which is obtained by feature extraction from a sample parking space image; the sample parking space image carries training labels; the training labels include parking space entrance key point labels, parking space entrance center point labels, parking space entrance direction labels, and position offset information labels.

[0042] The sample target feature image is input into the first branch network of the initial parking space entrance information extraction network. The first branch network predicts the key points of the sample parking space entrance based on the sample target feature image to obtain a heat map of the key points of the sample parking space entrance.

[0043] The sample target feature image is input into the second branch network of the initial parking space entrance information extraction network. The second branch network predicts the center point of the sample parking space entrance based on the sample target feature image to obtain a heat map of the center point of the sample parking space entrance.

[0044] The sample target feature image is input into the third branch network of the initial parking space entrance information extraction network. The third branch network predicts the sample parking space entrance direction information based on the sample target feature image to obtain the sample parking space entrance direction information.

[0045] The sample target feature image is input into the fourth branch network of the initial parking space entrance information extraction network. The fourth branch network predicts the sample position offset information based on the sample target feature image to obtain the sample position offset information.

[0046] Based on the heatmap of key points at the sample parking space entrance, the heatmap of the center point of the sample parking space entrance, the direction information of the sample parking space entrance, the sample position offset information, and the training labels, the parameters of each network layer in the initial parking space entrance information extraction network are adjusted until the training termination condition is met to obtain the parking space entrance information extraction network.

[0047] Optionally, the training labels further include location compensation information labels, and the training process further includes:

[0048] The sample target feature image is input into the fifth branch network of the initial parking space entrance information extraction network. The fifth branch network predicts the sample position compensation information based on the sample target feature image to obtain the sample position compensation information.

[0049] Based on the sample location compensation information and the location compensation information label, adjust the parameters of the fifth branch network in the initial parking space entrance information extraction network until the training termination condition is met to obtain the parking space entrance information extraction network.

[0050] Secondly, a parking space detection device is provided, the device comprising:

[0051] The target feature extraction module is used to acquire an image of a parking space to be detected, and to extract features from the image of the parking space to be detected to obtain a target feature image; the image of the parking space to be detected includes the entrance to at least one parking space;

[0052] The target parking space entrance information extraction module is used to predict parking space entrance information based on the target feature image to obtain target parking space entrance information. The target parking space entrance information includes a parking space entrance key point heatmap, a parking space entrance center point heatmap, parking space entrance direction information, and position offset information. The parking space entrance key point heatmap represents the confidence level that each pixel in the image of the parking space to be detected is a parking space entrance key point; the parking space entrance center point heatmap represents the confidence level that each pixel in the image of the parking space to be detected is a parking space entrance center point; the parking space entrance direction information represents the direction information corresponding to each pixel in the image of the parking space to be detected; and the position offset information represents the position offset amount corresponding to each pixel in the image of the parking space to be detected.

[0053] The target parking space entrance key point determination module is used to determine the target parking space entrance key point corresponding to each parking space entrance based on the parking space entrance key point heat map, the parking space entrance center point heat map and the position offset information;

[0054] The direction information determination module is used to determine the direction information corresponding to the key point of the target parking space entrance for each parking space entrance in the parking space entrance direction information;

[0055] The target parking space determination module is used to determine the target parking space based on the target parking space entrance key point corresponding to each parking space entrance and the direction information corresponding to the target parking space entrance key point.

[0056] Optionally, the target parking space entrance information extraction module includes:

[0057] The first branch network module is used to input the target feature image into the first branch network of the parking space entrance information prediction network, and the first branch network predicts the key points of the parking space entrance based on the target feature image to obtain the heat map of the key points of the parking space entrance.

[0058] The second branch network module is used to input the target feature image into the second branch network of the parking space entrance information prediction network, and the second branch network predicts the center point of the parking space entrance based on the target feature image to obtain the heat map of the center point of the parking space entrance.

[0059] The third branch network module is used to input the target feature image into the third branch network of the parking space entrance information prediction network, and the third branch network predicts the parking space entrance direction information based on the target feature image to obtain the parking space entrance direction information.

[0060] The fourth branch network module is used to input the target feature image into the fourth branch network of the parking space entrance information prediction network, and the fourth branch network predicts the position offset information based on the target feature image to obtain the position offset information.

[0061] The fifth branch network module is used to input the target feature image into the fifth branch network of the parking space entrance information prediction network, and the fifth branch network performs position compensation information prediction based on the target feature image to obtain the position compensation information.

[0062] Optionally, the target parking space entrance key point determination module includes:

[0063] The parking space entrance center point determination unit is used to obtain at least one parking space entrance center point based on the pixels in the parking space entrance center point heat map whose confidence level exceeds a preset confidence threshold; each parking space entrance center point corresponds to one parking space entrance.

[0064] The position offset determination unit is used to determine the position offset amount corresponding to the center point of each parking space entrance in the position offset information;

[0065] The first parking space entrance key point determination unit is used to determine the first parking space entrance key point associated with each parking space entrance center point based on the center point of each parking space entrance and the position offset corresponding to the center point of the parking space entrance.

[0066] The target parking space entrance key point determination unit is used to determine the target parking space entrance key point corresponding to each parking space entrance from a preset range centered on the first parking space entrance key point in the parking space entrance key point heat map.

[0067] Optionally, the target parking space entrance key point determination unit includes:

[0068] The second parking space entrance key point determination unit is used to obtain the second parking space entrance key point corresponding to each parking space entrance based on the pixel with the highest confidence level within a preset range centered on the first parking space entrance key point in the parking space entrance key point heat map.

[0069] A location compensation determination unit is used to determine the location compensation amount corresponding to the key point of the second parking space entrance for each parking space entrance in the location compensation information.

[0070] The key point correction unit is used to determine the target parking space entrance key point corresponding to each parking space entrance based on the second parking space entrance key point corresponding to each parking space entrance and the position compensation amount corresponding to the second parking space entrance key point.

[0071] Optionally, the target parking space determination module includes:

[0072] An angle calculation unit is used to determine an angle value based on the direction information corresponding to the key point of the target parking space entrance; the angle value represents the angle between the parking space entrance direction line connected to the key point of the target parking space entrance and the preset baseline in the clockwise direction;

[0073] A parking space entrance direction line determination unit is used to determine the parking space entrance direction line based on the angle value and the preset baseline.

[0074] The target parking space determination unit is used to determine the target parking space based on the parking space entrance direction line and the target parking space entrance key point.

[0075] Optionally, the target feature extraction module includes:

[0076] The feature extraction module is used to perform downsampling processing on the parking space image to be detected a preset number of times to obtain an initial feature image sequence; the initial feature image sequence includes multiple initial feature images of different resolutions;

[0077] The feature fusion module is used to fuse the multiple initial feature images of different resolutions in a preset order to obtain the target feature image.

[0078] Optionally, the feature fusion module includes:

[0079] An upsampling unit is used to take the initial feature image with the lowest resolution in the initial feature image sequence as the current image and perform upsampling processing on the current image;

[0080] A convolution unit is used to take the adjacent initial feature images of the current image in the initial feature image sequence as the first image in a preset order, and perform convolution processing on the first image of a preset size.

[0081] The feature fusion unit is used to perform feature fusion on the upsampled current image and the convolutionally processed first image to obtain the fused feature image;

[0082] The target feature image determination unit is used to update the current image based on the fused feature image until the initial feature image with the highest resolution in the initial feature image sequence is taken as the first image, and the resulting fused feature image is taken as the target feature image.

[0083] Optionally, the parking space detection device further includes a training module, the training module comprising:

[0084] The building unit is used to construct the initial parking space entrance information extraction network to be trained;

[0085] The sample acquisition unit is used to acquire a sample target feature image, which is obtained by feature extraction from a sample parking space image; the sample parking space image carries training labels; the training labels include parking space entrance key point labels, parking space entrance center point labels, parking space entrance direction labels, and position offset information labels.

[0086] The first branch network training unit is used to input the sample target feature image into the first branch network of the initial parking space entrance information extraction network, and the first branch network predicts the key points of the sample parking space entrance based on the sample target feature image to obtain a heat map of the key points of the sample parking space entrance.

[0087] The second branch network training unit is used to input the sample target feature image into the second branch network of the initial parking space entrance information extraction network, and the second branch network predicts the center point of the sample parking space entrance based on the sample target feature image to obtain a heat map of the center point of the sample parking space entrance.

[0088] The third branch network training unit is used to input the sample target feature image into the third branch network of the initial parking space entrance information extraction network, and the third branch network predicts the sample parking space entrance direction information based on the sample target feature image to obtain the sample parking space entrance direction information.

[0089] The fourth branch network training unit is used to input the sample target feature image into the fourth branch network of the initial parking space entrance information extraction network, and the fourth branch network predicts the sample position offset information based on the sample target feature image to obtain the sample position offset information.

[0090] The first parameter adjustment unit is used to adjust the parameters of each network layer in the initial parking space entrance information extraction network according to the sample parking space entrance key point heat map, the sample parking space entrance center point heat map, the sample parking space entrance direction information, the sample position offset information and the training label, until the training end condition is met to obtain the parking space entrance information extraction network.

[0091] Optionally, the training labels further include location compensation information labels, and the training module further includes:

[0092] The fifth branch network training unit is used to input the sample target feature image into the fifth branch network of the initial parking space entrance information extraction network, and the fifth branch network predicts the sample position compensation information based on the sample target feature image to obtain the sample position compensation information.

[0093] The second parameter adjustment unit is used to adjust the parameters of the fifth branch network in the initial parking space entrance information extraction network according to the sample position compensation information and the position compensation information label, until the training end condition is met to obtain the parking space entrance information extraction network.

[0094] Thirdly, a server is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the above-described parking space detection method.

[0095] Fourthly, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the parking space detection method as described above.

[0096] Fifthly, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the parking space detection method provided in the various optional implementations described above.

[0097] The technical solutions of the embodiments of the present invention have the following beneficial effects:

[0098] This invention constructs the key points of the target parking space entrance by extracting heatmaps of key points at the parking space entrance, heatmaps of the center point of the parking space entrance, parking space entrance direction information, and position offset information from the image of the parking space to be detected. This enables the detection of parking spaces even when the key points of the parking space in the image are incomplete. The key points of the parking space entrance are determined and corrected using the heatmap of the center point of the parking space entrance and the position offset information. At the same time, the direction information of the parking space entrance is predicted, making the parking space detection results more accurate. A lightweight end-to-end neural network is used to improve the network's operation and calculation speed. Attached Figure Description

[0099] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0100] Figure 1 This is a schematic diagram of the overall framework of a parking space detection method provided in an embodiment of the present invention;

[0101] Figure 2 This is a flowchart illustrating a parking space detection method provided in an embodiment of the present invention;

[0102] Figure 3 This is a schematic diagram of the framework of an optional method for implementing the parking space detection method provided in an embodiment of the present invention;

[0103] Figure 4 This is a schematic diagram of the parking space detection result of the implementation of the parking space detection method provided in the embodiment of the present invention;

[0104] Figure 5 This is a flowchart illustrating the training process of the parking space detection model provided in an embodiment of the present invention;

[0105] Figure 6a It is a heat map of key parking space points in existing technology;

[0106] Figure 6b This is a schematic diagram of a parking space detection label for a training parking space detection model provided in an embodiment of the present invention;

[0107] Figure 7a This is a structural block diagram of a parking space detection device provided in an embodiment of the present invention;

[0108] Figure 7b This is a structural block diagram of a parking space detection device in another optional method provided in an embodiment of the present invention;

[0109] Figure 8 This is a hardware structure block diagram of a server provided in an embodiment of the present invention. Detailed Implementation

[0110] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0111] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0112] With rapid urbanization and a significant increase in car ownership, the number of cars in cities is growing rapidly. Finding parking spaces during rush hour and holidays becomes extremely difficult, exacerbated by varying driver skill levels. However, with the development of artificial intelligence, automatic parking technology has made significant strides, making it possible for cars to automatically park in various complex environments. Automatic parking refers to parking without human intervention; the car uses perception and control algorithms to park itself automatically. Users simply press a button in the car or via a mobile app, and the vehicle automatically finds an available parking space and parks itself. This technology has garnered significant attention from major manufacturers, who are developing new technologies and algorithms to improve the accuracy of automatic parking. Parking space detection algorithms, as a prerequisite for automatic parking, are particularly crucial. Over the years, parking space detection algorithms have evolved from traditional computer vision methods to feature extraction using deep learning networks.

[0113] Common deep learning-based parking space detection methods include object detection, semantic segmentation, Transformer, or combinations of these methods. Object detection typically involves detecting parking space corners or bounding boxes; these methods treat parking space detection as a type of object detection. Semantic segmentation methods are complex in post-processing to obtain specific parking space information. Currently available Transformer (DETR) algorithms for parking space detection haven't significantly outperformed other methods, and their engineering implementation is more difficult.

[0114] Parking space detection methods based on key points utilize the principle that a fixed parking space can be determined by its four corner points. A neural network model is used to detect these four key points to identify the parking space. However, typical parking space detection images are obtained by stitching together images from a panoramic view. Due to the various stitching algorithms currently in use, the resulting images often show incomplete parking spaces (typically only half of the parking space including the entrance), meaning the four key points may not be complete. Therefore, to improve detection accuracy, this invention, in addition to detecting the two key points at the parking space entrance, simultaneously predicts the midpoint of the parking space entrance line and the direction of the entrance, overcoming the problems in existing technologies.

[0115] The parking space detection method provided in this invention is based on a neural network model implemented using machine learning. Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0116] See Figure 1 The diagram illustrates the overall framework of a parking space detection method provided by an embodiment of the present invention. The method comprises three parts: a backbone network model, a header network model, and a post-processing part. The backbone network model includes two network architectures, used for feature extraction from the parking space image to be detected. The header network model is used to extract the entrance information of each target parking space from the target feature image output by the backbone network model. The post-processing part is an algorithm independent of the neural network model, used to determine the target parking space based on the entrance information of each target parking space. Detailed steps are described below.

[0117] See Figure 2 The diagram illustrates a flowchart of a parking space detection method according to an embodiment of the present invention. It should be noted that while this specification provides the operational steps described in the embodiments or flowchart, more or fewer operational steps may be included based on conventional or non-inventive methods. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). The parking space detection method of this embodiment is implemented based on a parking space detection model, and the parking space detection method may include:

[0118] S201, acquire the image of the parking space to be detected, and extract features from the image of the parking space to be detected to obtain a target feature image; the image of the parking space to be detected includes the entrance of at least one parking space.

[0119] In one possible implementation, step S201 may include:

[0120] Specifically, step S201 corresponds to the backbone network model. In this embodiment, the backbone part is implemented using a MobilenetV2+FPN network architecture. MobilenetV2 (Depthwise Separable Convolutional Network) is a lightweight deep neural network proposed by Google for embedded devices such as mobile phones. Its core idea is depthwise separable convolution. In the process of image feature extraction through neural networks, the fewer channels, the less computational cost of multiplication in the convolutional layers. Therefore, if the entire network consists of low-dimensional channels, the overall computation speed will be fast; however, this is not ideal as it cannot extract enough overall information. Therefore, to achieve both high feature extraction and fast computation speed, the MobilenetV2 network can be used. The principle is to first expand the image dimensions, then extract features using depthwise separable convolution, and finally compress the data to reduce the network size. This invention uses the MobilenetV2 network as the backbone network, making the parking space detection computation process more lightweight and easier to implement in engineering.

[0121] like Figure 3 As shown, step (1) uses the MobilenetV2 network and takes the image of the parking space to be detected as input. The image of the parking space to be detected can be an image stitched together from the surround view images collected by the vehicle. In one embodiment, an image of the parking space to be detected is input into the MobilenetV2 network. The resolution is 512*512 and the number of channels is 3. The image of the parking space to be detected is downsampled by 4 times, 8 times, 16 times and 32 times in sequence to generate initial feature images S1, S2, S3 and S4 respectively. S1 has a resolution of 128*128 and a number of channels of 64; S2 has a resolution of 64*64 and a number of channels of 128; S3 has a resolution of 32*32 and a number of channels of 256; S4 has a resolution of 16*16 and a number of channels of 512.

[0122] Therefore, the parking space image to be detected is downsampled multiple times to obtain initial feature images of different resolutions, from bottom to top: low-level semantic feature images to high-level semantic feature images. The bottom-level initial feature images are larger in size, have fewer channels, and accurate target locations; they mainly contain low-level features of the parking space image to be detected (such as image edges and texture features). The high-level initial feature images are smaller in size, have more channels, and coarser target locations; they mainly contain high-level features of the image (such as key points at the parking space entrance and the center point of the parking space entrance).

[0123] (2) The target feature image is obtained by fusing the multiple initial feature images of different resolutions in a preset order.

[0124] In one possible implementation, step (2) may further include the step:

[0125] The initial feature image with the lowest resolution in the initial feature image sequence is taken as the current image, and the current image is upsampled.

[0126] According to a preset order, the adjacent initial feature images of the current image in the initial feature image sequence are taken as the first image, and the first image is subjected to convolution processing of a preset size;

[0127] The upsampled current image and the convolutionally processed first image are fused to obtain the fused feature image.

[0128] The current image is updated based on the fused feature image until the initial feature image with the highest resolution in the initial feature image sequence is taken as the first image, and the resulting fused feature image is taken as the target feature image.

[0129] For details, please refer to [link / reference]. Figure 3Step (2) employs an FPN network, also known as a feature pyramid network. As mentioned earlier, the initial feature images output by the MobilenetV2 network have different expressive capabilities at different levels. Shallow features mainly reflect details such as brightness and darkness, and edges, while deep features reflect a richer overall structure. Using shallow features alone cannot contain information about the overall structure, which weakens the expressive power of the features. Deep features are themselves constructed from shallow features, so they actually contain information about shallow features. If deep features are then fused into shallow features, both details and the overall structure are taken into account, and the fused features will have a richer expressive power. Therefore, the way to achieve this process is to use an FPN network, select several layers on the feature pyramid, and these layers themselves form a hierarchical relationship from shallow to deep. Then, the deep features are merged into the shallow layers step by step to form a new feature pyramid. Each layer of this new pyramid integrates information from both shallow and deep layers. By applying the features of each layer for detection, the purpose of detecting targets at different scales can be achieved.

[0130] The FPN network performs convolution processing on multiple initial feature images of different resolutions generated by the MobileNetV2 network to construct a feature pyramid. In this embodiment of the invention, an alignment operation is performed before the upper-layer and lower-layer features are added and fused. This ensures that the upper-layer features are fused to the correct positions in the lower-layer features, improving the network's accurate recognition capability. That is, the alignment operation not only performs channel alignment between the upper-layer and lower-layer features but also spatial alignment. From top to bottom, the upper-layer features are upsampled to increase their resolution. The upsampling convolution operation, also known as deconvolution, enlarges the input feature map, for example, by a factor of two. This is achieved by padding the input feature map with zeros according to a certain ratio, followed by normal convolution, which outputs a feature map larger than the input feature map, achieving the upsampling effect. The lower-layer features are then convolved to increase their number of channels.

[0131] For the two adjacent initial feature images S4 and S3, S4 is upsampled to increase its resolution, resulting in a feature image with a resolution of 32*32 and 512 channels, which is the current image. S3 originally has 256 channels, so it is subjected to two 1*1 convolutions to increase its channel count without changing its resolution, resulting in a feature image with a resolution of 32*32 and 512 channels, which is the first image. At this point, the current image and the first image have the same resolution and number of channels, but the current image contains richer semantic information than the first image. Therefore, the first image and the second image are element-wise added to achieve feature fusion, resulting in the fused feature image C3.

[0132] For the two adjacent initial feature images S3 and S2, since S3 already has a corresponding fused feature image C3, C3 is upsampled to increase its resolution, resulting in a feature image with a resolution of 64*64 and 512 channels, which is the current image. S2 originally has 128 channels, so it is subjected to four 1*1 convolutions to increase its channel count without changing its resolution, resulting in a feature image with a resolution of 64*64 and 512 channels, which is the first image. The current image and the first image are then element-wise added to achieve feature fusion, resulting in the fused feature image C2.

[0133] For the two adjacent initial feature images S2 and S1, since S2 already has a corresponding fused feature image C2, C2 is upsampled to increase its resolution, resulting in a feature image with a resolution of 128*128 and 512 channels, which is the current image. S1 originally has 64 channels, so it undergoes eight 1*1 convolutions to increase its channel count without changing its resolution, resulting in a feature image with a resolution of 128*128 and 512 channels, which is the first image. The current image and the first image are then element-wise added to achieve feature fusion, resulting in the fused feature image C1. After the above process, the fused feature images C1, C2, and C3 are finally obtained. C1, which incorporates all features from the initial feature images, is used as the target feature image and is used as the input to the header network model in the following steps to extract the entrance information of each target parking space.

[0134] The above implementation uses an FPN feature pyramid structure for feature fusion, which can distinguish simple targets using shallow features and complex targets using deep features. Furthermore, by fusing features from low and high layers of the neural network, it solves the problems of limited semantic information in low-level features and coarse target location information in high-level features. Similarly, for the detection of multi-scale targets, it uses dilated convolutions of different scales to achieve multiple receptive field branches, and detects targets of different scales on different branches. In this embodiment, all feature maps have multiple receptive fields, which not only further improves the detection accuracy of targets at different scales but also includes more information about the surrounding environment of the target.

[0135] S202, predict parking space entrance information based on the target feature image to obtain target parking space entrance information; the target parking space entrance information includes a parking space entrance key point heatmap, a parking space entrance center point heatmap, parking space entrance direction information, and position offset information; the parking space entrance key point heatmap represents the confidence level that each pixel in the parking space image to be detected is a parking space entrance key point, the parking space entrance center point heatmap represents the confidence level that each pixel in the parking space image to be detected is a parking space entrance center point; the parking space entrance direction information represents the direction information corresponding to each pixel in the parking space image to be detected; the position offset information represents the position offset amount corresponding to each pixel in the parking space image to be detected.

[0136] In one possible implementation, the target parking space entrance information further includes position compensation information, which characterizes the position compensation amount of each pixel in the image of the parking space to be detected.

[0137] In one possible implementation, step S202 includes:

[0138] (1) Input the target feature image into the first branch network of the parking space entrance information prediction network, and the first branch network performs parking space entrance key point prediction based on the target feature image to obtain the parking space entrance key point heat map.

[0139] (2) Input the target feature image into the second branch network of the parking space entrance information prediction network, and the second branch network predicts the center point of the parking space entrance based on the target feature image to obtain the heat map of the center point of the parking space entrance.

[0140] (3) Input the target feature image into the third branch network of the parking space entrance information prediction network, and the third branch network predicts the parking space entrance direction information based on the target feature image to obtain the parking space entrance direction information;

[0141] (4) Input the target feature image into the fourth branch network of the parking space entrance information prediction network, and the fourth branch network predicts the position offset information based on the target feature image to obtain the position offset information;

[0142] (5) Input the target feature image into the fifth branch network of the parking space entrance information prediction network, and the fifth branch network performs position compensation information prediction based on the target feature image to obtain the position compensation information.

[0143] Specifically, step S202 is implemented based on the header network model, which contains multiple convolutional modules. Each convolutional module extracts a corresponding target parking space entrance information. In this embodiment of the invention, the target feature image C1 is used as the input of the header. After convolutional processing by 5 branch networks, 5 target parking space entrance information are output, namely: heatmap, center, direction, reg, and offset.

[0144] The output includes the following dimensions: heatmap (128*128 resolution, 1 channel), center (128*128 resolution, 1 channel), direction (sine and cosine values), reg (position offset), offset information, offset amount in x and y directions, and offset compensation information.

[0145] S203, based on the heat map of the key points of the parking space entrance, the heat map of the center point of the parking space entrance, and the position offset information, determine the key points of the target parking space entrance corresponding to each parking space entrance.

[0146] In one possible implementation, step S203 may include the following steps:

[0147] (1) Based on the pixels in the heat map of the parking space entrance center point that have a confidence level exceeding a preset confidence threshold, at least one parking space entrance center point is obtained; each parking space entrance center point corresponds to one parking space entrance.

[0148] (2) Determine the position offset of the center point of each parking space entrance in the position offset information;

[0149] (3) Based on the center point of each parking space entrance and the position offset corresponding to the center point of the parking space entrance, determine the first parking space entrance key point associated with each parking space entrance center point;

[0150] Specifically, since the position offset information includes the offset of the parking space entrance key point relative to the parking space entrance center point, the offsets of the two parking space entrance key points associated with the current parking space entrance center point in the x and y directions are obtained from the position offset information output by reg. For example, if the coordinates of the parking space entrance center point C are (0,0), and the position offset information output by reg shows that the offset of point (0,0) in the x direction is +1 and the offset of point (0,0) in the y direction is +1, then the first initial position coordinates of the parking space entrance key point A associated with the parking space entrance center point C are (1,1); the other offset of point (0,0) in the x direction is -1 and the offset in the y direction is -1, then the first initial position coordinates of the parking space entrance key point B associated with the parking space entrance center point C are (-1,-1).

[0151] (4) From the heat map of key points of parking space entrances, within a preset range centered on the first key point of parking space entrance, determine the target key point of each parking space entrance.

[0152] In one possible implementation, step (4) may include:

[0153] Based on the pixel with the highest confidence level within a preset range centered on the first parking space entrance key point in the heat map of the parking space entrance key points, the second parking space entrance key point corresponding to each parking space entrance is obtained.

[0154] Determine the location compensation amount corresponding to the key point of the second parking space entrance for each parking space entrance in the location compensation information;

[0155] Based on the key point of the second parking space entrance corresponding to each parking space entrance and the position compensation amount corresponding to the key point of the second parking space entrance, the key point of the target parking space entrance corresponding to each parking space entrance is determined.

[0156] Specifically, the first parking space entrance key point is obtained from step (3). Then, the specific target parking space entrance key point is determined by combining the parking space entrance key point heat map output by the heatmap. In the parking space entrance key point heat map, a preset range is determined with the first parking space entrance key point as the center. For example, in one embodiment, a circle with a radius of 2 is set. Similar to the center point of the parking space entrance, the position with the highest confidence is found within this preset range of the parking space entrance key point heat map. The point at this position is taken as the second parking space entrance key point, which is more accurate than the first parking space entrance key point.

[0157] However, it's important to note that the branch feature image output by the header is a downsampled image (4 times the original image of the parking space to be detected). That is, the original image has a resolution of 512*512, while the header's output branch feature image has a resolution of 128*128. If points in the downsampled feature image are restored to their original resolution, there might be offset errors. For example, x=0 and x=3 at 512*512 resolution would both be x=0 at 128*128 resolution. Therefore, in the 128*128 resolution... The second initial position of the parking space entrance key point determined in the feature image, if it is (0,0), may not be (0,0) when restored to the original parking space image to be detected. It may be (0,3), (3,0), (3,3), etc. Therefore, this embodiment of the invention corrects this error by using offset to output the position compensation amount of the parking space entrance key point relative to itself in the x and y directions after downsampling feature extraction. Based on the second parking space entrance key point and the position compensation amount corresponding to the key point output by offset, a more accurate parking space entrance key point, i.e., the target parking space entrance key point, can be obtained.

[0158] S204, determine the direction information corresponding to the key point of the target parking space entrance for each parking space entrance in the parking space entrance direction information.

[0159] S205, Based on the key point of the target parking space entrance corresponding to each parking space entrance and the direction information corresponding to the key point of the target parking space entrance, determine the target parking space.

[0160] In one possible implementation, step S205 may include:

[0161] Based on the direction information corresponding to the key point of the target parking space entrance, an angle value is determined; the angle value represents the angle between the parking space entrance direction line connected to the key point of the target parking space entrance and the preset baseline in the clockwise direction;

[0162] The parking space entrance direction line is determined based on the angle value and the preset baseline.

[0163] The target parking space is determined based on the direction line of the parking space entrance and the key points of the target parking space entrance.

[0164] Specifically, since the acquired image of the parking space to be detected may not be directly facing the vehicle and may be deflected, the direction of the parking space entrance must be considered when detecting the parking space. Based on the sine and cosine values ​​output by the direction, the angle of the parking space entrance direction line connected to the key point of the target parking space entrance relative to the preset baseline in a clockwise direction is obtained. In this embodiment of the invention, the preset baseline is a vertical direction line coinciding with the 12 o'clock position. For example, if the coordinates of a key point A at the entrance of a target parking space are (1,1), the sine value corresponding to point (1,1) is obtained from the parking space entrance direction information. The cosine value is The angle between the parking space entrance direction line associated with key point A at the target parking space entrance and the preset baseline in a clockwise direction is 315 degrees, which means the parking space entrance direction line points 45 degrees to the upper left. Therefore, based on the key point and direction line at the target parking space entrance, the target parking space can be detected, and the detection result is as follows. Figure 4 As shown.

[0165] The training of the parking space detection model in this embodiment of the invention is described below, with reference to... Figure 5 The training process of the prediction network for the parking space entrance information may include the following steps:

[0166] S501, Construct the initial parking space entrance information extraction network to be trained.

[0167] Specifically, the initial parking space entrance information extraction network to be trained is as follows: Figure 1 The diagram shows the structure of the header network model in the overall framework diagram.

[0168] S502, acquire the sample target feature image, which is obtained by feature extraction from the sample parking space image; the sample parking space image carries training labels; the training labels include parking space entrance key point labels, parking space entrance center point labels, parking space entrance direction labels, and position offset information labels.

[0169] Specifically, the sample parking space images are input into the backbone network model, which outputs sample target feature images. These sample target feature images are then used as input to the initial parking space entrance information extraction network to be trained. Parking space entrance key point labels and parking space entrance center point labels are in the form of heatmaps. Parking space entrance direction labels and position offset information labels can be data annotations of the sample parking space images by professionals. In one embodiment, the parking space entrance key point labels can be a heatmap of directional parking space entrance key points, such as... Figure 6b As shown, its shape is elliptical, with the major axis indicating the direction of the parking space entrance, relative to... Figure 6aThe ordinary Gaussian kernel heatmap shown here has the orientation of the points added to represent the orientation of the parking space entrance.

[0170] S503, the sample target feature image is input into the first branch network of the initial parking space entrance information extraction network, and the first branch network predicts the key points of the sample parking space entrance based on the sample target feature image to obtain a heat map of the key points of the sample parking space entrance.

[0171] S504, the sample target feature image is input into the second branch network of the initial parking space entrance information extraction network, and the second branch network predicts the center point of the sample parking space entrance based on the sample target feature image to obtain a heat map of the center point of the sample parking space entrance.

[0172] S505, the sample target feature image is input into the third branch network of the initial parking space entrance information extraction network, and the third branch network predicts the sample parking space entrance direction information based on the sample target feature image to obtain the sample parking space entrance direction information.

[0173] S506, the sample target feature image is input into the fourth branch network of the initial parking space entrance information extraction network, and the fourth branch network predicts the sample position offset information based on the sample target feature image to obtain the sample position offset information.

[0174] S507, based on the heat map of key points at the sample parking space entrance, the heat map of the center point of the sample parking space entrance, the direction information of the sample parking space entrance, the sample position offset information, and the training label, adjust the parameters of each network layer in the initial parking space entrance information extraction network until the training termination condition is met to obtain the parking space entrance information extraction network.

[0175] In one possible implementation, the step may include: determining a loss value based on the differences between the sample parking space entrance key point heatmap, the sample parking space entrance center point heatmap, the sample parking space entrance direction information, the sample position offset information, and the training labels;

[0176] Based on the loss value, the parameters of each network layer in the initial parking space entrance information extraction network are adjusted until the preset training termination condition is met, thus obtaining the parking space detection model.

[0177] Specifically, a preset loss function can be used to calculate the loss value based on the output of each branch network and the training labels. Then, the parameters of each network layer in the initial parking space entrance information extraction network are adjusted in the direction of minimizing this loss value until the training termination condition is met, thus completing the training of the parking space entrance information extraction network. This training termination condition can be that the loss value reaches a preset loss threshold, or that the number of iterations reaches a preset number of iterations.

[0178] In one embodiment, the loss value of the first branch network is calculated based on the difference between the heatmap of the key points at the sample parking space entrance and the labels of the key points at the parking space entrance, and the parameters in the first branch network are adjusted in the direction that minimizes the loss value; the loss value of the second branch network is calculated based on the difference between the heatmap of the center point at the sample parking space entrance and the labels of the center point at the parking space entrance, and the parameters in the second branch network are adjusted in the direction that minimizes the loss value; the loss value of the third branch network is calculated based on the difference between the direction information of the sample parking space entrance and the direction label of the parking space entrance, and the parameters in the third branch network are adjusted in the direction that minimizes the loss value; the loss value of the fourth branch network is calculated based on the difference between the sample position offset information and the position offset information label, and the parameters in the fourth branch network are adjusted in the direction that minimizes the loss value.

[0179] In one possible implementation, the training label further includes a location compensation information label, and the training process further includes:

[0180] The sample target feature image is input into the fifth branch network of the initial parking space entrance information extraction network. The fifth branch network predicts the sample position compensation information based on the sample target feature image to obtain the sample position compensation information.

[0181] Based on the sample location compensation information and the location compensation information label, adjust the parameters of the fifth branch network in the initial parking space entrance information extraction network until the training termination condition is met to obtain the parking space entrance information extraction network.

[0182] Specifically, based on the difference between the sample location compensation information and the location compensation information label, the loss value of the fifth branch network is calculated, and the parameters in the fifth branch network are adjusted in the direction of minimizing this loss value.

[0183] Corresponding to the parking space detection methods provided in the above embodiments, this embodiment of the invention also provides a parking space detection device. Since the parking space detection device provided in this embodiment corresponds to the parking space detection methods provided in the above embodiments, the implementation methods of the aforementioned parking space detection methods are also applicable to the parking space detection device provided in this embodiment, and will not be described in detail in this embodiment.

[0184] Please see Figure 7a The diagram shows a schematic representation of a parking space detection device according to an embodiment of the present invention. This device has the function of implementing the parking space detection method described in the above-described method embodiments. This function can be implemented in hardware or by hardware executing corresponding software. Figure 7a As shown, the device may include:

[0185] The target feature extraction module 710 is used to acquire an image of a parking space to be detected and to extract features from the image of the parking space to be detected to obtain a target feature image; the image of the parking space to be detected includes the entrance to at least one parking space;

[0186] The target parking space entrance information extraction module 720 is used to predict parking space entrance information based on the target feature image to obtain target parking space entrance information. The target parking space entrance information includes a parking space entrance key point heatmap, a parking space entrance center point heatmap, parking space entrance direction information, and position offset information. The parking space entrance key point heatmap represents the confidence level that each pixel in the image of the parking space to be detected is a parking space entrance key point; the parking space entrance center point heatmap represents the confidence level that each pixel in the image of the parking space to be detected is a parking space entrance center point; the parking space entrance direction information represents the direction information corresponding to each pixel in the image of the parking space to be detected; and the position offset information represents the position offset amount corresponding to each pixel in the image of the parking space to be detected.

[0187] The target parking space entrance key point determination module 730 is used to determine the target parking space entrance key point corresponding to each parking space entrance based on the parking space entrance key point heat map, the parking space entrance center point heat map and the position offset information;

[0188] The direction information determination module 740 is used to determine the direction information corresponding to the key point of the target parking space entrance for each parking space entrance in the parking space entrance direction information.

[0189] The target parking space determination module 750 is used to determine the target parking space based on the target parking space entrance key point corresponding to each parking space entrance and the direction information corresponding to the target parking space entrance key point.

[0190] Optionally, the target parking space entrance information extraction module 720 includes:

[0191] The first branch network module is used to input the target feature image into the first branch network of the parking space entrance information prediction network, and the first branch network predicts the key points of the parking space entrance based on the target feature image to obtain the heat map of the key points of the parking space entrance.

[0192] The second branch network module is used to input the target feature image into the second branch network of the parking space entrance information prediction network, and the second branch network predicts the center point of the parking space entrance based on the target feature image to obtain the heat map of the center point of the parking space entrance.

[0193] The third branch network module is used to input the target feature image into the third branch network of the parking space entrance information prediction network, and the third branch network predicts the parking space entrance direction information based on the target feature image to obtain the parking space entrance direction information.

[0194] The fourth branch network module is used to input the target feature image into the fourth branch network of the parking space entrance information prediction network, and the fourth branch network predicts the position offset information based on the target feature image to obtain the position offset information.

[0195] The fifth branch network module is used to input the target feature image into the fifth branch network of the parking space entrance information prediction network, and the fifth branch network performs position compensation information prediction based on the target feature image to obtain the position compensation information.

[0196] Optionally, the target parking space entrance key point determination module 730 includes:

[0197] The parking space entrance center point determination unit is used to obtain at least one parking space entrance center point based on the pixels in the parking space entrance center point heat map whose confidence level exceeds a preset confidence threshold; each parking space entrance center point corresponds to one parking space entrance.

[0198] The position offset determination unit is used to determine the position offset amount corresponding to the center point of each parking space entrance in the position offset information;

[0199] The first parking space entrance key point determination unit is used to determine the first parking space entrance key point associated with each parking space entrance center point based on the center point of each parking space entrance and the position offset corresponding to the center point of the parking space entrance.

[0200] The target parking space entrance key point determination unit is used to determine the target parking space entrance key point corresponding to each parking space entrance from a preset range centered on the first parking space entrance key point in the parking space entrance key point heat map.

[0201] Optionally, the target parking space entrance key point determination unit includes:

[0202] The second parking space entrance key point determination unit is used to obtain the second parking space entrance key point corresponding to each parking space entrance based on the pixel with the highest confidence level within a preset range centered on the first parking space entrance key point in the parking space entrance key point heat map.

[0203] A location compensation determination unit is used to determine the location compensation amount corresponding to the key point of the second parking space entrance for each parking space entrance in the location compensation information.

[0204] The key point correction unit is used to determine the target parking space entrance key point corresponding to each parking space entrance based on the second parking space entrance key point corresponding to each parking space entrance and the position compensation amount corresponding to the second parking space entrance key point.

[0205] Optionally, the target parking space determination module 750 includes:

[0206] An angle calculation unit is used to determine an angle value based on the direction information corresponding to the key point of the target parking space entrance; the angle value represents the angle between the parking space entrance direction line connected to the key point of the target parking space entrance and the preset baseline in the clockwise direction;

[0207] A parking space entrance direction line determination unit is used to determine the parking space entrance direction line based on the angle value and the preset baseline.

[0208] The target parking space determination unit is used to determine the target parking space based on the parking space entrance direction line and the target parking space entrance key point.

[0209] Optionally, the target feature extraction module 710 includes:

[0210] The feature extraction module is used to perform downsampling processing on the parking space image to be detected a preset number of times to obtain an initial feature image sequence; the initial feature image sequence includes multiple initial feature images of different resolutions;

[0211] The feature fusion module is used to fuse the multiple initial feature images of different resolutions in a preset order to obtain the target feature image.

[0212] Optionally, the feature fusion module includes:

[0213] An upsampling unit is used to take the initial feature image with the lowest resolution in the initial feature image sequence as the current image and perform upsampling processing on the current image;

[0214] A convolution unit is used to take the adjacent initial feature images of the current image in the initial feature image sequence as the first image in a preset order, and perform convolution processing on the first image of a preset size.

[0215] The feature fusion unit is used to perform feature fusion on the upsampled current image and the convolutionally processed first image to obtain the fused feature image;

[0216] The target feature image determination unit is used to update the current image based on the fused feature image until the initial feature image with the highest resolution in the initial feature image sequence is taken as the first image, and the resulting fused feature image is taken as the target feature image.

[0217] Optional, see reference Figure 7b The parking space detection device further includes a training module 760, which comprises:

[0218] The building unit is used to construct the initial parking space entrance information extraction network to be trained;

[0219] The sample acquisition unit is used to acquire a sample target feature image, which is obtained by feature extraction from a sample parking space image; the sample parking space image carries training labels; the training labels include parking space entrance key point labels, parking space entrance center point labels, parking space entrance direction labels, and position offset information labels.

[0220] The first branch network training unit is used to input the sample target feature image into the first branch network of the initial parking space entrance information extraction network, and the first branch network predicts the key points of the sample parking space entrance based on the sample target feature image to obtain a heat map of the key points of the sample parking space entrance.

[0221] The second branch network training unit is used to input the sample target feature image into the second branch network of the initial parking space entrance information extraction network, and the second branch network predicts the center point of the sample parking space entrance based on the sample target feature image to obtain a heat map of the center point of the sample parking space entrance.

[0222] The third branch network training unit is used to input the sample target feature image into the third branch network of the initial parking space entrance information extraction network, and the third branch network predicts the sample parking space entrance direction information based on the sample target feature image to obtain the sample parking space entrance direction information.

[0223] The fourth branch network training unit is used to input the sample target feature image into the fourth branch network of the initial parking space entrance information extraction network, and the fourth branch network predicts the sample position offset information based on the sample target feature image to obtain the sample position offset information.

[0224] The first parameter adjustment unit is used to adjust the parameters of each network layer in the initial parking space entrance information extraction network according to the sample parking space entrance key point heat map, the sample parking space entrance center point heat map, the sample parking space entrance direction information, the sample position offset information and the training label, until the training end condition is met to obtain the parking space entrance information extraction network.

[0225] Optionally, the training labels further include location compensation information labels, and the training module 760 further includes:

[0226] The fifth branch network training unit is used to input the sample target feature image into the fifth branch network of the initial parking space entrance information extraction network, and the fifth branch network predicts the sample position compensation information based on the sample target feature image to obtain the sample position compensation information.

[0227] The second parameter adjustment unit is used to adjust the parameters of the fifth branch network in the initial parking space entrance information extraction network according to the sample position compensation information and the position compensation information label, until the training end condition is met to obtain the parking space entrance information extraction network.

[0228] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0229] This invention provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the parking space detection method provided in the above method embodiments.

[0230] The memory can be used to store software programs and modules. The processor executes various functional applications and parking space detection by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0231] The methods and embodiments provided in this invention can be executed on a computer terminal, server, or similar computing device. Taking running on a server as an example... Figure 8 This is a hardware structure block diagram of a server for running a parking space detection method provided in an embodiment of the present invention, as shown below. Figure 8As shown, the server 800 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 810 (CPUs 810 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 830 for storing data, and one or more storage media 820 (e.g., one or more mass storage devices) for storing application programs 823 or data 822. The memory 830 and storage media 820 may be temporary or persistent storage. The program stored in the storage media 820 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 810 may be configured to communicate with the storage media 820 and execute the series of instruction operations stored in the storage media 820 on the server 800. Server 800 may also include one or more power supplies 860, one or more wired or wireless network interfaces 850, one or more input / output interfaces 840, and / or one or more operating systems 821, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0232] The input / output interface 840 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 800. In one example, the input / output interface 840 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 840 may be a radio frequency (RF) module for wireless communication with the Internet.

[0233] Those skilled in the art will understand that Figure 8 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 800 may also include... Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown.

[0234] Embodiments of the present invention also provide a computer-readable storage medium, which may be disposed in a server to store at least one instruction or at least one program related to implementing a parking space detection method, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the parking space detection method provided in the above-described method embodiments.

[0235] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0236] Embodiments of the present invention also provide a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the parking space detection method provided in the various optional implementations described above.

[0237] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0238] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0239] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0240] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A parking space detection method, characterized in that, include: Acquire an image of the parking space to be detected, and extract features from the image to obtain a target feature image; The image of the parking space to be detected includes the entrance to at least one parking space; Based on the target feature image, the parking space entrance information is predicted by the parking space entrance information prediction network to obtain the target parking space entrance information; the target parking space entrance information includes a heat map of key points of the parking space entrance, a heat map of the center point of the parking space entrance, parking space entrance direction information and position offset information; The heatmap of key points at the parking space entrance represents the confidence level that each pixel in the image of the parking space to be detected is a key point at the parking space entrance, and the heatmap of the center point at the parking space entrance represents the confidence level that each pixel in the image of the parking space to be detected is the center point at the parking space entrance. The parking space entrance direction information represents the direction information corresponding to each pixel in the image of the parking space to be detected. The position offset information represents the position offset of each pixel in the image of the parking space to be detected relative to the center point of the corresponding parking space entrance; Based on the pixels in the heat map of the parking space entrance center point that have a confidence level exceeding a preset confidence threshold, at least one parking space entrance center point is obtained; each parking space entrance center point corresponds to one parking space entrance. Determine the position offset of the center point of each parking space entrance in the position offset information; Based on the center point of each parking space entrance and the position offset corresponding to the center point of the parking space entrance, determine the first parking space entrance key point associated with each of the parking space entrance center points; From the heat map of key points at parking space entrances, within a preset range centered on the first key point at parking space entrance, determine the target key point at each parking space entrance; Determine the directional information corresponding to the key points of the target parking space entrance for each parking space entrance in the parking space entrance directional information; The target parking space is determined based on the key point of the target parking space entrance corresponding to each parking space entrance and the direction information corresponding to the key point of the target parking space entrance.

2. The parking space detection method according to claim 1, characterized in that, The prediction of parking space entrance information based on the target feature image to obtain the target parking space entrance information includes: The target feature image is input into the first branch network of the parking space entrance information prediction network. The first branch network predicts the key points of the parking space entrance based on the target feature image to obtain the heat map of the key points of the parking space entrance. The target feature image is input into the second branch network of the parking space entrance information prediction network. The second branch network predicts the center point of the parking space entrance based on the target feature image to obtain the heat map of the center point of the parking space entrance. The target feature image is input into the third branch network of the parking space entrance information prediction network, and the third branch network predicts the parking space entrance direction information based on the target feature image to obtain the parking space entrance direction information. The target feature image is input into the fourth branch network of the parking space entrance information prediction network, and the fourth branch network predicts the position offset information based on the target feature image to obtain the position offset information.

3. The parking space detection method according to any one of claims 1 or 2, characterized in that, The target parking space entrance information also includes position compensation information, which represents the position compensation amount of each pixel in the image of the parking space to be detected. The step of extracting the parking space entrance information based on the target feature image to obtain the target parking space entrance information further includes: The target feature image is input into the fifth branch network of the parking space entrance information prediction network, and the fifth branch network predicts the location compensation information based on the target feature image to obtain the location compensation information. The step of determining the target parking space entrance key point corresponding to each parking space entrance within a preset range centered on the first parking space entrance key point from the heat map of the parking space entrance key points includes: Based on the pixel with the highest confidence level within a preset range centered on the first parking space entrance key point in the heat map of the parking space entrance key points, the second parking space entrance key point corresponding to each parking space entrance is obtained. Determine the location compensation amount corresponding to the key point of the second parking space entrance for each parking space entrance in the location compensation information; Based on the key point of the second parking space entrance corresponding to each parking space entrance and the position compensation amount corresponding to the key point of the second parking space entrance, the key point of the target parking space entrance corresponding to each parking space entrance is determined.

4. The parking space detection method according to claim 1, characterized in that, The step of determining the target parking space based on the key point of the target parking space entrance and the direction information corresponding to the key point of the target parking space entrance includes: Based on the direction information corresponding to the key point of the target parking space entrance, an angle value is determined; the angle value represents the angle between the parking space entrance direction line connected to the key point of the target parking space entrance and the preset baseline in the clockwise direction; The parking space entrance direction line is determined based on the angle value and the preset baseline. The target parking space is determined based on the direction line of the parking space entrance and the key points of the target parking space entrance.

5. The parking space detection method according to claim 1, characterized in that, The step of extracting features from the image of the parking space to be detected to obtain the target feature image includes: The image of the parking space to be detected is downsampled a preset number of times to obtain an initial feature image sequence; the initial feature image sequence includes multiple initial feature images of different resolutions; The target feature image is obtained by fusing features from the multiple initial feature images of different resolutions in a preset order.

6. The parking space detection method according to claim 5, characterized in that, The step of fusing features from multiple initial feature images of different resolutions in a preset order to obtain the target feature image includes: The initial feature image with the lowest resolution in the initial feature image sequence is taken as the current image, and the current image is upsampled. According to a preset order, the adjacent initial feature images of the current image in the initial feature image sequence are taken as the first image, and the first image is subjected to convolution processing of a preset size; The upsampled current image and the convolutionally processed first image are fused to obtain the fused feature image. The current image is updated based on the fused feature image until the initial feature image with the highest resolution in the initial feature image sequence is taken as the first image, and the resulting fused feature image is taken as the target feature image.

7. The parking space detection method according to claim 2, characterized in that, It also includes a training process for the prediction network of the parking space entrance information, the training process including: Construct an initial parking space entrance information extraction network to be trained; A sample target feature image is obtained, which is obtained by feature extraction from a sample parking space image; the sample parking space image carries training labels; the training labels include parking space entrance key point labels, parking space entrance center point labels, parking space entrance direction labels, and position offset information labels. The sample target feature image is input into the first branch network of the initial parking space entrance information extraction network. The first branch network predicts the key points of the sample parking space entrance based on the sample target feature image to obtain a heat map of the key points of the sample parking space entrance. The sample target feature image is input into the second branch network of the initial parking space entrance information extraction network. The second branch network predicts the center point of the sample parking space entrance based on the sample target feature image to obtain a heat map of the center point of the sample parking space entrance. The sample target feature image is input into the third branch network of the initial parking space entrance information extraction network. The third branch network predicts the sample parking space entrance direction information based on the sample target feature image to obtain the sample parking space entrance direction information. The sample target feature image is input into the fourth branch network of the initial parking space entrance information extraction network. The fourth branch network predicts the sample position offset information based on the sample target feature image to obtain the sample position offset information. Based on the heatmap of key points at the sample parking space entrance, the heatmap of the center point of the sample parking space entrance, the direction information of the sample parking space entrance, the sample position offset information, and the training labels, the parameters of each network layer in the initial parking space entrance information extraction network are adjusted until the training termination condition is met to obtain the parking space entrance information extraction network.

8. The parking space detection method according to claim 7, characterized in that, The training labels also include location compensation information labels, and the training process further includes: The sample target feature image is input into the fifth branch network of the initial parking space entrance information extraction network. The fifth branch network predicts the sample position compensation information based on the sample target feature image to obtain the sample position compensation information. Based on the sample location compensation information and the location compensation information label, adjust the parameters of the fifth branch network in the initial parking space entrance information extraction network until the training termination condition is met to obtain the parking space entrance information extraction network.

9. A parking space detection device, characterized in that, include: The target feature extraction module is used to acquire the image of the parking space to be detected and to extract the features of the image of the parking space to be detected to obtain the target feature image; The image of the parking space to be detected includes the entrance to at least one parking space; The target parking space entrance information extraction module is used to predict the target parking space entrance information based on the target feature image through a parking space entrance information prediction network to obtain the target parking space entrance information. The target parking space entrance information includes a heat map of key points at the parking space entrance, a heat map of the center point of the parking space entrance, parking space entrance direction information, and location offset information; The heatmap of key points at the parking space entrance represents the confidence level that each pixel in the image of the parking space to be detected is a key point at the parking space entrance, and the heatmap of the center point at the parking space entrance represents the confidence level that each pixel in the image of the parking space to be detected is the center point at the parking space entrance. The parking space entrance direction information represents the direction information corresponding to each pixel in the image of the parking space to be detected. The position offset information represents the position offset of each pixel in the image of the parking space to be detected relative to the center point of the corresponding parking space entrance; The parking space entrance center point determination unit is used to obtain at least one parking space entrance center point based on the pixels in the parking space entrance center point heat map whose confidence level exceeds a preset confidence threshold; each parking space entrance center point corresponds to one parking space entrance. The position offset determination unit is used to determine the position offset amount corresponding to the center point of each parking space entrance in the position offset information; The first parking space entrance key point determination unit is used to determine the first parking space entrance key point associated with each parking space entrance center point based on the center point of each parking space entrance and the position offset corresponding to the center point of the parking space entrance. The target parking space entrance key point determination unit is used to determine the target parking space entrance key point corresponding to each parking space entrance from a preset range centered on the first parking space entrance key point in the parking space entrance key point heat map; The direction information determination module is used to determine the direction information corresponding to the key point of the target parking space entrance for each parking space entrance in the parking space entrance direction information; The target parking space determination module is used to determine the target parking space based on the target parking space entrance key point corresponding to each parking space entrance and the direction information corresponding to the target parking space entrance key point.

10. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the steps of the method as claimed in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the steps of the method as claimed in any one of claims 1 to 8.

Images