A single-stage parking space detection system and method based on surround view images

Abstract

The application discloses a single-stage parking space detection system and method based on a ring image, comprising: a backbone network used for feature extraction on an input image, outputting a feature map; a bottleneck network used for fusing local features and global features in a feature vector contained in the feature map output by the backbone network through convolution, pooling, activation and SE attention mechanism, and splicing the local features and the global features into a multi-channel feature layer; an anchor point setting module used for dividing a picture of the multi-channel feature layer into multiple grids; a detection head network used for predicting a parking space result for each anchor point; and a decoding module used for decoding the parking space result predicted by the detection head network. The application can eliminate information redundancy in an existing three-stage parking space detection scheme, and simultaneously fuse an entry point matching and a parking space orientation stage designed artificially with entry point detection, so that the parking space detection is optimized into a single-stage scheme.

Description

Technical Field

[0001] This invention relates to the field of autonomous parking technology, and in particular to a single-stage parking space detection system and method based on surround view images. Background Technology

[0002] Parking space detection is a crucial component of autonomous parking systems, as its output determines the effectiveness of subsequent autonomous parking path planning and decision-making. Mainstream parking space detection solutions divide the task into three stages: entrance point detection, entrance point matching, and category and direction determination.

[0003] The entry points are the two vertices of the entry line. For example... Figure 1 As shown, Figure 1 The left and right points are the entrance points, while the left point is located inside the parking space, facing the entrance line of that parking space. Figure 1 When connecting the left and right points (the line connecting the middle left and right points), the entry point is on the left.

[0004] The entry point detection method varies depending on the entry point modeling method. For example, when the entry point modeling method is rectangular, the entry point detection method extracts features from entry points at different rotation angles based on the axis parallel to the rectangle. This results in inconsistent entry point features, and the parking spaces at different rotation angles contain a lot of background information during entry point matching, thus affecting the network detection performance. Another example is when the entry point modeling method is directional point modeling. The entry point detection method requires a manually designed matching template, but it can only handle vertical and horizontal parking spaces. Once the shape of the parking space entry point changes, the matching template becomes unsuitable.

[0005] The above-mentioned entrance detection methods all share a common problem: they detect two entrances belonging to the same parking space separately without considering their overall integrity, and the extracted features have serious redundancy. Summary of the Invention

[0006] The purpose of this invention is to provide a single-stage parking space detection system and method based on surround view images to overcome or at least mitigate at least one of the above-mentioned defects of the prior art.

[0007] To achieve the above objectives, the present invention provides a single-stage parking space detection system based on surround view images, comprising:

[0008] The backbone network is used to extract features from the input image and output a feature map. The feature map contains the extracted feature vectors, including local features and global features. Local features include the location of the parking space entrance, and global features include parking space occupancy and category.

[0009] The bottleneck network is used to fuse the local and global features in the feature vectors contained in the feature map output by the backbone network through convolution, pooling, activation, and SE attention mechanisms, and concatenate them into a multi-channel feature layer for output.

[0010] The anchor point setting module receives the multi-channel feature layer output from the bottleneck network. The image of this multi-channel feature layer is divided into multiple grids, each with a width and height of w. G and h G The coordinates of the anchor point are the top-left corner of each grid cell, represented in a pixel coordinate system with the top-left corner of the image as the origin as (x...). G y G );

[0011] The detection head network receives the multi-channel feature layer with anchors output by the anchor point setting module, and predicts a parking space result for each anchor point. The parking space result includes the parking space category result, whether the parking space is occupied result, the distance confidence result of the parking space, and the coordinates of the left point (x, y) and the offsets of the right point (w, h) relative to the left point in the pixel coordinate system with the top left corner of the image as the origin.

[0012] The decoding module is used to decode the parking space result predicted by the detection head network according to the following formula (1), and output the x-coordinate of the left point of the parking space entry line. left and the vertical coordinate y left The horizontal coordinate offset Δx and vertical coordinate offset Δy of the right point of the parking space entry line relative to the left point:

[0013]

[0014] Where, x pred y pred These represent the predicted values ​​output by the detection head network, corresponding to the x, y, and x coordinates of the left point of the parking space, respectively. max and Y max These are the maximum X-coordinate difference and Y-coordinate difference between the left and right points of the parking spaces in the data set collected by the data acquisition unit. The specific values ​​of these two values ​​are determined based on the distribution of parking spaces in the images taken by the acquisition vehicle. σ represents the Sigmoid activation function, and tanh represents the activation function.

[0015] Furthermore, the single-stage parking space detection system based on surround view images also includes:

[0016] Anchor label assignment module: This module is used to assign labels to each anchor point during network training. Specifically, it includes:

[0017] a cost calculation unit configured to calculate a cost between each parking space ground truth, to obtain a cost matrix, wherein the cost to be calculated includes: a category cross-entropy cost, a weighted straight-line distance cost L distance ;

[0018]

[0019] wherein, is a predicted coordinate of a left point of a parking space entry line by the parking space detection model, corresponding to x left in formula (1) left , is a predicted coordinate of a right point of a parking space entry line by the parking space detection model, corresponding to x left +Δx and y left +Δy in formula (1) respectively, is a ground truth of a left point coordinate of a parking space entry line, is a ground truth of a right point coordinate of a parking space entry line, and λ is a weight balance coefficient of the distance loss cost of the left point and the right point;

[0020] a normalized distance loss calculation unit configured to take the first 5 anchor points with the smallest L distance in each ground truth and , add the normalized distance loss of the anchor points, and take the integer part to obtain the number k of positive samples to be assigned for each ground truth;

[0021] a positive sample assignment unit configured to take the first k anchor points with the lowest sum of all costs in the ground truth as the positive samples of the ground truth.

[0022] Further, the backbone network has an hourglass network, which is divided into a preset number n of steps, wherein the first step is used to perform convolution processing on the input image, and then input the processed image to the other n-1 steps for convolution processing, and each of the other steps outputs a feature map containing a feature vector.

[0023] Further, the bottleneck network has n-1 CBR units, SE units and a channel splicing unit, wherein:

[0024] Each CBR has the same number as the other n-1 steps and corresponds to one of the other n-1 steps, and each CBR is configured to receive the feature vector contained in the feature map output by one of the other n-1 steps, perform convolution processing on the input, output a feature layer after feature extraction, and then pass through a corresponding SE unit for channel attention mechanism processing, output a feature layer after attention mechanism processing, and finally output a multi-channel feature layer after channel splicing processing by the channel splicing unit.

[0025] Further, the detection head network has n-1 full connection detection head branches and one convolution head detection branch, wherein:

[0026] Each full connection detection head branch is used for respectively receiving the multi-channel feature layer with anchor points, and converting the multi-channel feature layer from a feature domain to a result domain, the first full connection detection head branch outputs a parking space category result, the second full connection detection head branch outputs a parking space occupancy result, and the third full connection detection head branch outputs a parking space distance confidence result;

[0027] The convolution head detection branch is used for receiving the multi-channel feature layer with anchor points, and converting the multi-channel feature layer from a feature domain to a result domain, and outputting a parking space left point coordinate x, y in a pixel coordinate system with the upper left corner of an image as an origin, and a right point offset w and h relative to the left point.

[0028] The application further provides a single-stage parking space detection system based on a surround view image, which comprises:

[0029] A data acquisition unit composed of fisheye cameras mounted around a vehicle, used for acquiring a surround view image;

[0030] A data labeling unit used for labeling a parking space in the acquired surround view image by using a labeling tool, and the information to be labeled includes an entrance point, a tail point of the parking space, and whether the parking space is occupied;

[0031] A model training unit used for modeling two entrance points of a parking space as a directed line segment from a left point to a right point of the parking space, and training the single-stage parking space detection system based on the surround view image by using the labeled information;

[0032] A preprocessing unit used for scaling and standardizing the surround view image acquired by the data acquisition unit, and aligning a down-sampling rate of the parking space detection model;

[0033] A model inference unit used for detecting and recognizing the parking space position information in the preprocessed image by using the single-stage parking space detection system based on the surround view image trained by the model training unit, and giving the post-processing unit the information in a pixel coordinate format;

[0034] A post-processing unit used for converting the pixel coordinates to a world coordinate system according to calibrated parameters.

[0035] The application further provides a single-stage parking space detection method based on a surround view image, which comprises:

[0036] Step 1, feature extraction is performed on the input image, and a feature map is output, which contains the extracted feature vectors, including local features and global features, the local features include the entrance point position of the parking space, and the global features include the parking space occupation and category;

[0037] Step 2, the feature vectors contained in the feature map output by the backbone network are fused by convolution, pooling, activation and SE attention mechanism, and the local features and global features in the feature vectors are spliced into a multi-channel feature layer and output;

[0038] Step 3, receiving the multi-channel feature layer output by the bottleneck network, the picture of the multi-channel feature layer is divided into multiple grids, and the width and height of each grid are w G and h G , the coordinates of the anchor points are at the top left corner of each grid, and are represented as (x G , y G ) in the pixel coordinate system with the top left corner of the image as the origin;

[0039] Step 4, receiving the multi-channel feature layer with anchor points output by the anchor point setting module, predicting a parking space result for each anchor point, the parking space result includes the parking space category result, the parking space occupation result, the distance confidence result, and the left point coordinates x and y of the parking space in the pixel coordinate system with the top left corner of the image as the origin, and the offset w and h of the right point relative to the left point.

[0040] Step 5, decoding the parking space result predicted by the detection head network according to the following formula (1) to output the horizontal coordinate x left and the vertical coordinate y left of the left point of the parking space entry line, and the horizontal coordinate offset Δx and the vertical coordinate offset Δy of the right point of the parking space entry line relative to the left point:

[0041]

[0042] Wherein, x pred , y pred respectively represent the predicted values output by the detection head network, which respectively correspond to the left point coordinates x, y of the parking space, X max and Y max are the maximum X coordinate difference and Y coordinate difference of the left point and the right point of the parking space in the data set collected by the data acquisition unit, the specific values of the two are determined according to the distribution of the parking space in the picture shot by the collection vehicle, σ represents the Sigmoid activation function, and tanh represents the activation function.

[0043] Further, the single-stage parking space detection method based on the surround view image further comprises:

[0044] Step 6, assigning a label to each anchor point when training the neural network, specifically including:

[0045] Step 61, calculate the cost between each parking space true value, get the cost matrix, which needs to calculate the cost contains: category cross entropy cost, weighted straight distance cost L distance ;

[0046]

[0047] wherein, is the predicted coordinate of the left point of the parking space entry line by the parking space detection model, corresponding to x left in formula (1) left , is the predicted coordinate of the right point of the parking space entry line by the parking space detection model, corresponding to x left + Δx and y left + Δy in formula (1) respectively, is the true value of the manually labeled coordinate of the left point of the parking space entry line, is the true value of the manually labeled coordinate of the right point of the parking space entry line, and λ is the weight balance coefficient of the distance loss cost of the left point and the right point;

[0048] Step 62, take the first 5 anchor points with the smallest L distance in each true value and , add the normalized distance loss to get the number k of positive samples that need to be assigned for each true value;

[0049] Step 63, take the first k anchor points with the lowest sum of all costs in the true value as the positive samples of the true value.

[0050] Further, step 2 specifically comprises:

[0051] Each CBR receives the feature vectors contained in the feature maps output by the other n-1 steps respectively, and performs convolution processing on the input, outputs the feature layer after feature extraction, and then respectively performs channel attention mechanism processing through the corresponding SE unit, outputs the feature layer after attention mechanism processing, and finally is summarized to the channel splicing unit for channel splicing processing, and outputs the multi-channel feature layer.

[0052] Further, step 4 specifically comprises:

[0053] Step 41, each full connection detection head branch receives the multi-channel feature layer with anchor points respectively, and converts the multi-channel feature layer from the feature domain to the result domain, the first full connection detection head branch outputs the parking space category result, the second full connection detection head branch outputs the parking space occupation result, and the third full connection detection head branch outputs the distance confidence result of the parking space.

[0054] Step 42, the branch receiving the multi-channel feature layer with anchor points through the convolution head, and converting the multi-channel feature layer from the feature domain to the result domain, outputting the left point coordinates x, y of the parking space and the offset w and h of the right point relative to the left point in the pixel coordinate system with the upper left corner of the image as the origin.

[0055] The present application has the following advantages due to the above technical solutions:

[0056] The present application can eliminate the information redundancy in the existing three-stage parking space detection scheme, and simultaneously integrate the manually designed entry point matching and the parking space orientation stage with the entry point detection, to optimize the parking space detection into a single-stage scheme. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a schematic diagram of a parking space.

[0058] Figure 2 is a schematic diagram of a parking space type.

[0059] Figure 3 is a structural schematic diagram of a parking space detection system provided by an embodiment of the present application.

[0060] Figure 4 is a structural diagram of a single-stage parking space detection model based on a surround view image in the present application. Figure 3 DETAILED DESCRIPTION

[0061] The present application will be described in detail below in combination with the drawings and embodiments.

[0062] As shown in Figure 3 , the single-stage parking space detection system based on a surround view image provided by an embodiment of the present application includes a data acquisition unit, a data preprocessing unit, a data labeling unit, a model training unit, a model inference unit and a post-processing unit.

[0063] The data acquisition unit is composed of fisheye cameras mounted around the vehicle, and the fisheye cameras are characterized by wide field of view, which are used to acquire surround view images.

[0064] The data labeling unit is used to label the parking space in the acquired surround view image by using a labeling tool, and the information to be labeled includes the entry point, the tail point of the parking space and whether the parking space is occupied.

[0065] The model training unit is used to model the two entry points of the parking space as a directed line segment from the left point to the right point of the parking space, and train the single-stage parking space detection model based on the surround view image by using the labeled information.

[0066] ​The preprocessing unit is configured to perform scaling and normalization on the surround view image collected by the data collection unit, and to align the down-sampling rate of the parking space detection model.

[0067] The model inference unit is configured to use the parking space detection model trained by the model training unit to detect and identify the parking space position information in the preprocessed image, and to output the information in the format of pixel coordinates to the post-processing unit.

[0068] The post-processing unit is configured to convert the pixel coordinates into the world coordinate system according to the calibrated parameters.

[0069] As shown in Figure 4 The single-stage parking space detection model based on the surround view image provided by the embodiment of the application includes a backbone network (Hourglass), a bottleneck network (Neck), an anchor point setting module, a detection head network (Head), and a decoding module. Figure 4 In particular,

[0070] The HourGlass represents the backbone network, which is configured to extract features from the preprocessed image of the preset size, and output three feature maps containing the extracted feature vectors, including local features and global features, wherein the local features include the entrance point position of the parking space, and the global features include the occupancy and category of the parking space.

[0071] Specifically, the backbone network has a HourGlass module, which is divided into n stages, Figure 3 In particular, n=4, and the stages are: Stage1 represents the first stage, Stage2 represents the second stage, Stage3 represents the third stage, and Stage4 represents the fourth stage. The first stage is configured to perform convolution processing on the input image, and then input the processed image to the second stage, the third stage, and the fourth stage for convolution processing, respectively, to output one feature map containing a feature vector from each stage.

[0072] The preset size may be (3, 608, 608) for example, wherein the numbers in the brackets respectively represent the channel number, the length, and the width of the image. It should be noted that the length and the width of the image may be set to be multiples of the down-sampling multiple (32 times) of the trained neural network. The size of the output feature map is related to the size of the input image, the channel number is fixed, and the length and the width are respectively one thirty-second, one sixteenth, and one eighth of the input image. For example, the sizes of the three output feature maps are (256, 19, 19), (256, 38, 38), and (256, 76, 76) respectively.

[0073] Neck represents a bottleneck network, which is used to fuse local features and global features in feature vectors contained in feature maps output by the last three of the four stages of the backbone network through convolution, pooling, activation, and SE attention mechanism, splice them into a multi-channel feature layer, and output.

[0074] Specifically, the bottleneck network has n-1 CBR (Convolution, Batch normalization, ReLU) units, an SE (Squeeze-and-Excitation) unit, and a channel splicing (C) unit, where:

[0075] Each CBR is used to respectively receive feature vectors contained in feature maps output by the second, third, and fourth stages, and perform convolution processing on the input to output a feature layer after feature extraction, then respectively pass through the corresponding SE unit for channel attention mechanism processing to output a feature layer after attention mechanism processing, and finally aggregate to the channel splicing unit for channel splicing processing to output a multi-channel feature layer.

[0076] In this embodiment, the bottleneck network adopts a structure different from FPN, and outputs multiple feature maps of the backbone network to one level after passing through the SE attention module, which greatly reduces the number of sample outputs and alleviates the imbalance between positive and negative samples.

[0077] The anchor point setting module is used to receive the multi-channel feature layer output by the bottleneck network. The image of the multi-channel feature layer is divided into multiple grid cells, each grid has a width w and a height h, the coordinates of the anchor points are at the top left corner of each grid, and are represented as (x, y) in the pixel coordinate system with the top left corner of the image as the origin. G and h G G G

[0078] Head represents a detection head network, which is used to receive the multi-channel feature layer with anchor points output by the anchor point setting module, predict a parking space result for each anchor point, and the parking space result contains a parking space category result (Cls), such as a right angle, an acute angle, and an obtuse angle, and also includes a parking space occupancy result (vacant), a parking space distance confidence result (dis), and pixel coordinates x and y of the left point of the parking space in the pixel coordinate system with the top left corner of the image as the origin, as well as the offset w and h of the right point relative to the left point. The distance confidence result (dis) of the parking space can represent the accuracy of the parking space position prediction.

[0079] ​​​Specifically, the detection head network has n-1 full connection detection head branches (FC-Head) and one convolution head detection branch (Conv-Head), wherein:

[0080] Each full connection detection head branch is configured to receive the multi-channel feature layer with the anchor point and convert the multi-channel feature layer from a feature domain to a result domain, and the first full connection detection head branch outputs a parking space category result (Cls), the second full connection detection head branch outputs a parking space occupancy result (vacant), and the third full connection detection head branch outputs a parking space distance confidence result (dis). In the embodiment, the full connection detection head branch uses different weights in different regions of the image, and is more sensitive to space and can better distinguish the local and global of the target object.

[0081] The convolution head detection branch is configured to receive the multi-channel feature layer with the anchor point and convert the multi-channel feature layer from the feature domain to the result domain, and output the parking space left point coordinates x and y in the pixel coordinate system with the upper left corner of the image as the origin and the offset w and h of the right point relative to the left point. In the embodiment, the convolution head detection branch is more suitable for detecting the boundary of an object, and a branch Conv Head composed of convolution layers is used to solve the regression problem. In order to reflect the accuracy of the parking space position regression, the normalized distance between straight lines is additionally predicted.

[0082] The decoding module is configured to decode the parking space result predicted by the detection head network according to the following formula (1) to output the horizontal coordinate x left and the vertical coordinate y left of the left point of the parking space entry line and the horizontal coordinate offset Δx and the vertical coordinate offset Δy of the right point of the parking space entry line relative to the left point:

[0083]

[0084] wherein x pred and y pred represent the predicted values output by the detection head network, i.e., correspond to the left point coordinates x and y of the parking space, respectively, X max and Y max are the maximum X coordinate difference and Y coordinate difference between the left point and the right point of the parking space in the data set collected by the data acquisition unit, the specific values of which are determined according to the distribution of the parking space in the picture taken by the vehicle, and σ represents the Sigmoid activation function and tanh represents the activation function.

[0085] In one embodiment, the parking space detection model provided by the embodiment of the present application further comprises an anchor point label assignment module, which is configured to assign a label to each anchor point when training the network, and specifically comprises:

[0086] The cost calculation unit is used to calculate the cost between the true value of each parking space and the cost matrix. The costs to be calculated include: category cross-entropy cost and weighted straight-line distance cost (see equation (2) below). The category cross-entropy cost is calculated using -ln(p), and the weighted straight-line distance cost L distance The result is obtained using the following formula (2).

[0087]

[0088] in, The parking space detection model predicts the coordinates of the left point of the parking space entry line, corresponding to x in Formula 1. left and y left , The parking space detection model predicts the coordinates of the right point of the parking space entry line, corresponding to x in Formula 1. left +Δx and y left +Δy, This represents the true coordinates of the left point of the manually marked parking space entry line. λ represents the true coordinates of the right point of the parking space entry line marked manually. λ is the weight balance coefficient of the distance loss cost between the left and right points, which is a constraint on the left point of the parking space entry line. When choosing a specific value for λ, for example, try to select the value with the best effect in turn, such as 0.1, 0.2, ..., 0.9.

[0089] The normalized distance loss calculation unit is used to retrieve each true value. and L in distance The five smallest anchor points are summed up with their normalized distance losses and rounded down to obtain the number of positive samples k that each ground truth (GT) needs to be allocated. Here, a "positive sample" can be understood as the parking space detection network predicting a parking space at each anchor point, which is a sample. If the predicted parking space is a real parking space, then the sample is a positive sample.

[0090] The positive sample allocation unit is used to select the k anchors with the lowest sum of costs among all true values ​​as positive samples of that true value, and then use this to allocate positive samples to all true values.

[0091] This invention also provides a single-stage parking space detection method based on a surround view image, comprising:

[0092] Step 1: Extract features from the input image and output a feature map. The feature map contains the extracted feature vectors, including local features and global features. Local features include the location of the parking space entrance, and global features include parking space occupancy and category.

[0093] Step 2, the feature vectors contained in the feature maps output by the backbone network are fused by convolution, pooling, activation and SE attention mechanism to obtain a multi-channel feature layer, and the multi-channel feature layer is output;

[0094] Step 3, receiving the multi-channel feature layer output by the bottleneck network, the image in the multi-channel feature layer is divided into a plurality of grids, and the width and height of each grid are w G and h G respectively, the coordinates of the anchor points are at the top left corner of each grid, and are represented as (x G , y G ) in a pixel coordinate system with the top left corner of the image as the origin;

[0095] Step 4, receiving the multi-channel feature layer with anchor points output by the anchor point setting module, predicting a parking space result for each anchor point, the parking space result including a parking space category result, a parking space occupancy result, a distance confidence result of the parking space, and pixel coordinates x and y of a left point of the parking space in a pixel coordinate system with the top left corner of the image as the origin, and a horizontal coordinate offset w and a vertical coordinate offset h of a right point relative to the left point;

[0096] Step 5, decoding the parking space result predicted by the detection head network according to the above formula (1) to output the horizontal coordinate x left and the vertical coordinate y left of the left point of the parking space entry line, and the horizontal coordinate offset Δx and the vertical coordinate offset Δy of the right point of the parking space entry line relative to the left point.

[0097] In one embodiment, the single-stage parking space detection method based on the surround view image further comprises:

[0098] Step 6, assigning a label to each anchor point when training the neural network, specifically including:

[0099] Step 61, calculating the cost between each parking space true value to obtain a cost matrix, wherein the cost to be calculated includes: category cross-entropy cost, weighted straight-line distance cost L distance , as shown in the above formula (2);

[0100] Step 62, taking the first 5 anchor points with the smallest L distance in each true value and , adding the normalized distance loss of the anchor points to obtain the number k of positive samples to be assigned to each true value by taking the integer part;

[0101] Step 63, taking the first k anchor points with the lowest sum of all costs in the true value as the positive samples of the true value.

[0102] In one embodiment, step 2 specifically includes:

[0103] The feature vectors contained in the feature maps output by the other n-1 steps are respectively received by each CBR, and convolution processing is performed on the input, to output a feature layer after feature extraction, then the corresponding SE unit is respectively used to perform channel attention mechanism processing, to output a feature layer after attention mechanism processing, and finally, the channel splicing unit is used to perform channel splicing processing, to output a multi-channel feature layer.

[0104] In one embodiment, step 4 specifically comprises:

[0105] Step 41: the multi-channel feature layer with anchors is respectively received by each full connection detection head branch, and the multi-channel feature layer is converted from a feature domain to a result domain, the first full connection detection head branch outputs a parking space category result, the second full connection detection head branch outputs a parking space occupancy result, and the third full connection detection head branch outputs a parking space distance confidence result.

[0106] Step 42: the multi-channel feature layer with anchors is received by the convolution head detection branch, and the multi-channel feature layer is converted from a feature domain to a result domain, to output a parking space left point coordinate x, y in a pixel coordinate system with the upper left corner of an image as an origin, and a right point offset w and h relative to the left point.

[0107] The single-stage parking space detection scheme of the present application models the parking space as a whole, without the need for an entry point matching and direction judgment process, maximizes the use of the parking space information provided by the target detection network, optimizes the entire parking space detection process, and improves the speed and accuracy.

[0108] Finally, it should be pointed out that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them. Those skilled in the art should understand that the technical solutions described in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A single-stage parking space detection system based on surround view images, characterized in that, include: The backbone network is used to extract features from the input image and output a feature map. The feature map contains the extracted feature vectors, including local features and global features. Local features include the location of the parking space entrance, and global features include parking space occupancy and category. The bottleneck network is used to fuse the local and global features in the feature vectors contained in the feature map output by the backbone network through convolution, pooling, activation, and SE attention mechanisms, and concatenate them into a multi-channel feature layer for output. The anchor point setting module receives the multi-channel feature layer output from the bottleneck network. The image of this multi-channel feature layer is divided into multiple grids, each with a width and height of w. G and h G The coordinates of the anchor point are the top-left corner of each grid cell, represented in a pixel coordinate system with the top-left corner of the image as the origin as (x...). G y G ); The detection head network is used to receive the multi-channel feature layer with anchors output by the anchor setting module, and predict a parking space result for each anchor point. The parking space result includes the parking space category result, whether the parking space is occupied result, the distance confidence result of the parking space, and the left point coordinates x and y and the right point offset w and h relative to the left point in the pixel coordinate system with the top left corner of the image as the origin. The decoding module is used to decode the parking space result predicted by the detection head network according to the following formula (1), and output the x-coordinate of the left point of the parking space entry line. left and the vertical coordinate y left The horizontal coordinate offset Δx and vertical coordinate offset Δy of the right point of the parking space entry line relative to the left point: Where, x pred y pred These represent the predicted values ​​output by the detection head network, corresponding to the x, y, and x coordinates of the left point of the parking space, respectively. max and Y max These are the maximum X-coordinate difference and Y-coordinate difference between the left and right points of the parking space in the data set collected by the data acquisition unit. The specific values ​​of these two values ​​are determined based on the distribution of parking spaces in the images taken by the acquisition vehicle. σ represents the Sigmoid activation function, and tanh represents the activation function. The bottleneck network has n-1 CBR units, SE units, and channel splicing units, where: The number of CBRs is the same as the number of the other n-1 steps, and they correspond one-to-one. Each CBR is used to receive the feature vector contained in the feature map output by one of the other n-1 steps, and performs convolution processing on the input. The output is a feature layer after feature extraction. Then, it is processed by the channel attention mechanism through the corresponding SE unit, and the output is a feature layer processed by the attention mechanism. Finally, it is summarized to the channel concatenation unit, which performs channel concatenation processing to output a multi-channel feature layer.

2. The single-stage parking space detection system based on surround view images as described in claim 1, characterized in that, Also includes: Anchor label assignment module: This module is used to assign labels to each anchor point during network training. Specifically, it includes: The cost calculation unit is used to calculate the cost between the ground truth value of each parking space and the cost matrix. The costs to be calculated include: category cross-entropy cost and weighted straight-line distance cost L. distance ; in, The predicted coordinates of the left point of the parking space entry line by the parking space detection model are x in equation (1). left and y left , The predicted coordinates of the parking space detection model for the right point of the parking space entry line are respectively x in equation (1). left +Δx and y left +Δy, This represents the true coordinates of the left point of the manually marked parking space entry line. λ represents the true coordinates of the right point of the manually marked parking space entry line, and λ is the weighting balance coefficient of the distance loss cost between the left and right points. The normalized distance loss calculation unit is used to retrieve each true value. and L in distance The five smallest anchor points are summed up by their normalized distance losses and rounded down to obtain the number of positive samples k that each true value needs to be assigned. Positive sample allocation unit, which is used to take the top k anchors with the lowest sum of all costs in the true value as positive samples of the true value.

3. The single-stage parking space detection system based on surround view images as described in claim 1 or 2, characterized in that, The backbone network is an hourglass network, which is divided into a preset number of n steps. The first step is used to perform convolution processing on the input image, and then the image is fed into the other n-1 steps for convolution processing. The other steps each output a feature map, which contains feature vectors.

4. The single-stage parking space detection system based on surround view images as described in claim 1, characterized in that, The detection head network has n-1 fully connected detection head branches and one convolutional detection head branch, where: Each fully connected detection head branch is used to receive the multi-channel feature layer with anchor points and convert the multi-channel feature layer from the feature domain to the result domain. The first fully connected detection head branch outputs the parking space category result, the second fully connected detection head branch outputs the parking space occupancy result, and the third fully connected detection head branch outputs the parking space distance confidence result. The convolutional head detection branch is used to receive the multi-channel feature layer with anchor points and convert the multi-channel feature layer from the feature domain to the result domain, outputting the coordinates x and y of the left point of the parking space and the offset w and h of the right point relative to the left point in the pixel coordinate system with the top left corner of the image as the origin.

5. A single-stage parking space detection method based on surround view images, characterized in that, include: Step 1: Extract features from the input image and output a feature map. The feature map contains the extracted feature vectors, including local features and global features. Local features include the location of the parking space entrance, and global features include parking space occupancy and category. Step 2: The feature vectors contained in the feature map output by the backbone network are fused into a multi-channel feature layer by using convolution, pooling, activation, and SE attention mechanisms. The fusion of local and global features in the feature vectors is then performed and output. Step 3: Receive the multi-channel feature layer output from the bottleneck network. The image of this multi-channel feature layer is divided into multiple grids, each with a width and height of w. G and h G The coordinates of the anchor point are the top-left corner of each grid cell, represented in a pixel coordinate system with the top-left corner of the image as the origin as (x...). G y G ); Step 4: Receive the multi-channel feature layer with anchors output by the anchor point setting module, predict a parking space result for each anchor point. The parking space result includes the parking space category result, whether the parking space is occupied result, the distance confidence result of the parking space, and the left point coordinates x and y and the right point offset w and h relative to the left point in the pixel coordinate system with the top left corner of the image as the origin. Step 5: Decode the parking space prediction result from the detection head network according to the following formula (1) and output the x-coordinate of the left point of the parking space entry line. left and the vertical coordinate y left The horizontal coordinate offset Δx and vertical coordinate offset Δy of the right point of the parking space entry line relative to the left point: Where, x pred y pred These represent the predicted values ​​output by the detection head network, corresponding to the x, y, and x coordinates of the left point of the parking space, respectively. max and Y max These are the maximum X-coordinate difference and Y-coordinate difference between the left and right points of the parking space in the data set collected by the data acquisition unit. The specific values ​​of these two values ​​are determined based on the distribution of parking spaces in the images taken by the acquisition vehicle. σ represents the Sigmoid activation function, and tanh represents the activation function. Step 2 specifically includes: Each CBR receives the feature vector contained in the feature map output by one of the other n-1 steps, performs convolution processing on the input, outputs a feature layer after feature extraction, then performs channel attention mechanism processing through the corresponding SE unit, outputs a feature layer after attention mechanism processing, and finally summarizes to the channel concatenation unit, which performs channel concatenation processing to output a multi-channel feature layer.

6. The single-stage parking space detection method based on surround view image as described in claim 5, characterized in that, Also includes: Step 6: Assign a label to each anchor point during neural network training, specifically including: Step 61: Calculate the cost between the ground truth value and the parking space, obtaining a cost matrix. The costs to be calculated include: category cross-entropy cost and weighted straight-line distance cost L. distance ; in, The predicted coordinates of the left point of the parking space entry line by the parking space detection model are x in equation (1). left and y left , The predicted coordinates of the parking space detection model for the right point of the parking space entry line are respectively x in equation (1). left +Δx and y left +Δy, This represents the true coordinates of the left point of the manually marked parking space entry line. λ represents the true coordinates of the right point of the manually marked parking space entry line, and λ is the weighting balance coefficient of the distance loss cost between the left and right points. Step 62, take each truth value and L in distance The five smallest anchor points are summed up by their normalized distance losses and rounded down to obtain the number of positive samples k that each true value needs to be assigned. Step 63: Take the k anchor points with the lowest sum of all costs in the true value as positive samples of the true value.

7. The single-stage parking space detection method based on surround view image as described in claim 5, characterized in that, Step 4 specifically includes: Step 41: Receive the multi-channel feature layer with anchor points through each fully connected detection head branch, and convert the multi-channel feature layer from the feature domain to the result domain. The parking space category result is output by the first fully connected detection head branch, the parking space occupancy result is output by the second fully connected detection head branch, and the distance confidence result of the parking space is output by the third fully connected detection head branch. Step 42: Receive the multi-channel feature layer with anchor points through the detection branch of the convolution head, and convert the multi-channel feature layer from the feature domain to the result domain. Output the coordinates x and y of the left point of the parking space and the offset w and h of the right point relative to the left point in the pixel coordinate system with the top left corner of the image as the origin.

Images