A deep learning-based parking space and obstacle detection method and application thereof

Abstract

The application discloses a kind of based on deep learning's parking space and obstacle detection method and its application, the method includes:1.in the corresponding position of vehicle installation fish eye camera;2.to fish eye camera is looked around splicing and is labeled, data set is made;3.build neural network structure, adopt a main network is shared, after feature extraction network, generate the network structure of two detection heads, respectively, different ways of regression are carried out to parking space and obstacle;4.train neural network, obtain the weight under optimal loss, carry out parking space identification.The application avoids the problem that parking space cannot be fitted in the process of simultaneously detecting parking space and obstacle by using bounding box regression mode, so as to ensure that parking space can be accurately fitted while obstacle can be effectively detected and identified.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent vehicle perception, in particular to a parking space and obstacle detection method based on deep learning and application thereof. BACKGROUND

[0002] At present, intelligent driving has become a mainstream trend, and the automatic parking function is one of the most basic and important functions in intelligent driving. In the face of tight parking spaces and complex parking environments, due to the existence of parking blind area, it is difficult for the driver to timely find pedestrians and obstacles in the blind area, and it is also difficult to accurately estimate the steering angle and reversing distance of the car, which will affect the parking safety and operation.

[0003] The traditional automatic parking assistance system mainly relies on the distance prompt of ultrasonic radar, and the application scene is limited, lacks visual information, the position information is fuzzy, the driver needs to consider driving and environmental observation, the attention is scattered, and the parking success rate is low. The existing network model simply detects parking space information and ignores obstacle information, so as to perceive the obstacles in the surrounding environment such as pedestrians, bicycles, and ground locks, resulting in partial missing of the perceived environmental information, and the driver still needs to observe the surrounding environment of the vehicle body to scatter attention and reduce the parking success rate. SUMMARY

[0004] In order to solve the above problems of the prior art, the present application provides a parking space and obstacle detection method based on deep learning and application thereof, so as to detect parking spaces and obstacles during automatic parking, and ensure the accuracy of environmental perception during automatic parking.

[0005] In order to achieve the above application purposes, the present application adopts the following technical solutions:

[0006] The parking space and obstacle detection method based on deep learning comprises the following steps:

[0007] Step 1, splicing of the surround view image and preprocessing of the data set:

[0008] Step 1.1, fish-eye cameras are installed at the front bumper, rear license plate, and lower ends of the left and right rearview mirrors of the vehicle, used to obtain images collected by each fish-eye camera, and the images are respectively corrected for distortion, perspective transformation, and spliced into a surround view image, and the surround view image is optimized by overlapping area feathering to obtain an optimized surround view spliced image;

[0009] Step 1.2, obtain a surround view stitching image set of the vehicle driving near the parking space in the manner of step 1.1, and label the parking space in the i-th surround view stitching image, if the parking space in the i-th surround view stitching image is at the edge, take the corner point of the parking space at the edge as the origin, and make two directional ray arrows to form a directional marker arrow of the "L" shape of the parking space; if the parking space is at a non-edge position, take the common corner point of the parking space as the origin, and make three directional ray arrows to form a directional marker arrow of the "T" shape; thereby obtaining the n-th parking space labeling information in the i-th surround view stitching image, including: the origin coordinates the shape type of the directional marker arrow and the rotation angle of the directional marker arrow

[0010] The obstacles in the i-th surround view stitching image are also labeled, and the m-th obstacle labeling information in the i-th surround view stitching image includes: the category of the obstacle the center point coordinates of the bounding box where the obstacle is located the width the height Further, the labeled surround view stitching image set U = u1, u2, …, u i …, u I}∈R H×W×X , wherein u i represents the i-th surround view stitching image containing labeling information; H represents the height of the surround view stitching image, W represents the width of the surround view stitching image, and C represents the number of channels of the surround view stitching image; I represents the total number of surround view stitching images; let u i The parking space labeling information of u i is denoted as L i , and the obstacle labeling information of u i is denoted as H q ;

[0011] Step 2, build a convolutional neural network based on deep learning, which sequentially includes: a backbone network module, a pyramid pooling structure, a feature prediction module, a parking space detection module Head1, and an obstacle detection module Head2:

[0012] Step 2.1, the backbone network module is composed of Q down-sampling blocks and P residual blocks, and the Q down-sampling blocks are respectively DSampleBlock1, …, DSampleBlock q , …, DSampleBlock Q ; wherein DSampleBlock q represents the q-th down-sampling block, q = 1, 2, …, Q;

[0013] The qth downsampling block DSampleBlock q by the qth multi-layer two-dimensional convolutional layer Dconv2D q and the qth downsampling layer DSample q ; wherein the qth multi-layer two-dimensional convolutional layer Dconv2D q comprises X two-dimensional convolutional layers, X batch normalization layers and X LeakyReLU activation function layers, wherein the convolution kernel size of the xth two-dimensional convolutional layer is k x ×k x ;

[0014] P residual blocks are ResBlock1,…,ResBlock p ,…,ResBlock P ; wherein ResBlock p represents the pth residual block, p = 1, 2,…, P;

[0015] The pth residual block ResBlock p is composed of r multi-layer two-dimensional convolutional layers Resconv2D, and the first multi-layer two-dimensional convolution Resconv2D p in the pth residual block ResBlock P,1 is connected to the rth multi-layer two-dimensional convolution Resconv2D P,r ;

[0016] Any one of the labeled panoramic stitching images u i in the labeled panoramic stitching image set U is input into the deep learning convolutional neural network, and is processed by the Q downsampling blocks and P residual blocks of the backbone network in turn to obtain a feature map DResult i ∈R H×W×C′ ; D' represents the channel number of the feature map DResult i processed by the backbone network;

[0017] Step 2.2, input the feature map DResult i into the pyramid pooling structure, first perform three maximum pooling operations respectively to obtain a first pooling feature map MaxpoolDResult i,1 , a second pooling feature map MaxpoolDResult i,2 , and a third pooling feature map MaxpoolDResult i,3 , and then perform channel splicing on the three pooling feature maps and DResult i to output a pyramid pooling feature map DSPPResult i ∈R H×W×C”; C" represents the number of channels of the pyramid pooling feature map;

[0018] Step 2.3, the feature prediction module is composed of a channel attention mechanism unit SE and branch 1 and branch 2;

[0019] The branch 1 is composed of k multi-layer two-dimensional convolution layers Neckconv2D1, …, Neckconv2D k , wherein the kth multi-layer two-dimensional convolution layer Neckconv2D k includes Y two-dimensional convolution layers, Y batch normalization layers and Y LeakyReLU activation function layers, wherein the convolution kernel size of the yth two-dimensional convolution layer is k y × k y ;

[0020] The branch 2 is composed of a residual module NeckResblock and a two-dimensional convolution layer conv2D, and the residual module NeckResblock is composed of l multi-layer two-dimensional convolution layers NeckResbconv2D1, …, NeckResconv2D l , wherein the lth two-dimensional convolution layer NeckResconv2D l includes Z two-dimensional convolution layers, Z batch normalization layers and Z LeakyReLU activation function layers, wherein the convolution kernel size of the zth two-dimensional convolution layer is k' z × k' z ;

[0021] The pyramid pooling feature map DSPPResult i is input into the feature prediction module, and the channel attention mechanism unit SE is used to perform global average pooling processing to obtain an average pooling feature map AavgDSPPResult i with a dimension of (H, W, C"); i After the average pooling feature map AavgDSPPResult i is compressed into a feature vector with a dimension of (1, 1, C"), the channel dimension of the feature vector is reduced to (1, 1, C" / r) through the first fully connected layer FC1, and then processed through the Swish activation function, and then through the second fully connected layer FC2, the dimension of the feature vector is raised back to (1, 1, C"), and finally converted into a normalized weight vector between 0 and 1 through the sigmoid function; after the normalized weight vector and DSPPResult i are multiplied channel by channel, a weighted attention mechanism feature map SEDSPPResult H×W×C” ∈R H×W×C” ;

[0022] Step 2.4: Transfer the attention mechanism feature map SEDSPPResult i The input branch 1 is processed by convolution to obtain a feature map PslotResult containing parking space detection information. i ∈R H×W×C”′ C”′ represents the number of channels in the feature map containing parking space detection information;

[0023] The weighted feature map SEDSPPResult i The input is processed in branch 2 and undergoes l convolutions to obtain the feature map NeckResResult. l Then use SEDSPPResult i With NeckResResult l The feature map NeckConcatResult is obtained after skip stitching. i Then, the feature map NeckConcatResult is processed. i After processing with a 2D convolutional layer (conv2D), the result is obtained.

[0024] Feature map DetectResult containing obstacle detection information i ∈RH ×W×C”” "C" represents the number of channels in the feature map containing obstacle detection information;

[0025] The feature map PslotResult containing parking space detection information i Input the predicted parking space information into the parking space detection module Head1. Among them, parking space information The information of the nth directional arrow is Let be the predicted coordinates of the origin of the nth directional arrow in the i-th image. Let n be the predicted value of the category information of the nth directional arrow in the i-th image. It is L-shaped when It is T-shaped; Let be the predicted cosine and sine values ​​of the nth directional marker arrow in the i-th image, respectively. Let n be the confidence level of the nth directional arrow in the i-th image;

[0026] The feature map DetectResult containing obstacle detection information i The data is fitted into the Head2 obstacle detection module to obtain the predicted obstacle information. Among them, obstacle information The information of the m-th obstacle bounding box is is the prediction value of the center coordinate of the mth obstacle bounding box in the ith picture, is the prediction value of the width and height of the mth obstacle bounding box in the ith picture, is the category information of the obstacle detected by the mth bounding box in the ith picture;

[0027] Step 2.5, establishing a back propagation loss function:

[0028] Information of the directional marker arrow The first loss function Loss1 is obtained by performing mean square error loss calculation;

[0029] Positioning information of the obstacle bounding box Performing mean square error loss calculation, and The second loss function Loss2 is obtained by performing binary cross entropy loss calculation; and the total loss function Loss is calculated total = Loss1 + Loss2.

[0030] Step 3, inputting the surround stitching image set U into a deep learning-based convolutional neural network for training, and calculating the total loss function Loss total When the number of training iterations reaches the set number of times, the training is stopped, so that the optimal parking space and obstacle extraction network is obtained, which is used for identifying the parking space and obstacles in automatic parking.

[0031] The electronic device comprises a memory and a processor, wherein the memory is used for storing a program supporting the processor to execute the parking space and obstacle detection method, and the processor is configured to execute the program stored in the memory.

[0032] The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to execute the steps of the parking space and obstacle detection method.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] 1. The present application proposes a parking space and obstacle detection method based on deep learning, builds a neural network structure for identifying parking spaces and obstacles in the automatic parking process, solves the problem that only parking spaces can be identified but obstacles cannot be identified in the traditional network, and solves the problem that the corners of the parking space are roughly identified by using the bounding box method, causing inaccurate positioning.

[0035] 2、The neural network structure provided by the application belongs to a shallow network structure in deep learning, can be carried on a vehicle-mounted embedded development board, has low requirements on the performance of a computing platform, and has low costs for dataset production and training. The convolutional neural network based on deep learning is mature in algorithm and is easy to transplant. The common main network is adopted, two feature extraction network structures are utilized to generate two detection modules, and the detection of parking spaces and obstacles is realized. Different regression methods are adopted for the detection of parking spaces and obstacles, the point and line regression is adopted for the detection of parking spaces, the method of surrounding frame regression in YOLO is adopted for the detection of obstacles, so that the detection result can be more clearly and intuitively displayed. Meanwhile, the detection accuracy of parking spaces and obstacles can be improved by adopting different regression methods. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The application is a kind of overall flow chart of a parking space and obstacle detection method based on deep learning;

[0037] Figure 2 The application is a channel attention module structure diagram;

[0038] Figure 3 The application is an overall network structure diagram. DETAILED DESCRIPTION

[0039] In the embodiment, a kind of parking space and obstacle detection method based on deep learning is used Four fisheye cameras are spliced into a surround view image, and a kind of improved neural network is proposed by using the method of deep learning, which can detect parking space and obstacle information simultaneously, and different regression methods are used, which can judge the category of obstacle and well fit the contour of parking space at the same time, specifically, as shown in Figure 1 As shown in the figure, the following steps are included:

[0040] Step 1, splicing of surround view image and data set preprocessing:

[0041] Step 1.1, fisheye camera is installed at the front bumper position, rear license plate position, left and right rearview mirror lower end position of the vehicle; according to the images collected by each camera, the camera intrinsic matrix K1, K2, K3 and K4 are obtained by using Zhang Zhengyou calibration method and MABTLAB calibration tool;

[0042] Select 4 pairs of 3D points in space and their corresponding pixel points, use PnP algorithm to obtain the external parameters of each camera M1 = [R1|T1], M2 = [R2|T2], M3 = [R3|T3], M4 = [R4|T4], use the imaging of the checkerboard in the fisheye camera, mark 4 points on the source image as (x1, x2), (x3, x4), (x5, x6), (x7, x8), these 4 points define a rectangle, use OpenCV tools to set the size of the above rectangle as the pixel size of the image, and the four corner points as the pixel coordinates of the four corner points, corresponding to (x'1, x'2), (x'3, x'4), (x'5, x'6), (x'7, x'8), the perspective transformation matrix can be obtained by the formula Do the same operation for the other three fisheye cameras to obtain P2, P3, P4, and use the obtained perspective transformation matrix to perform perspective transformation on the image to obtain the overhead view image, and obtain the overhead view of each view.

[0043] Step 1.2, obtain the surround view splicing image set of the vehicle driving near the parking space in the manner of step 1.1, and label the parking space in the i-th surround view splicing image, if the parking space in the i-th surround view splicing image is at the edge, take the corner point of the parking space at the edge as the origin, and draw two directional arrowheads in two directions to form a directional marker arrow of "L" shape; if the parking space is not at the edge, take the common corner point of the parking space as the origin, and draw three directional arrowheads to form a directional marker arrow of "T" shape; thereby obtaining the n-th parking space labeling information in the i-th surround view splicing image, including: origin coordinates the shape type of the directional marker arrow and the rotation angle of the directional marker arrow

[0044] Label the obstacles in the i-th surround view splicing image to obtain the m-th obstacle labeling information in the i-th surround view splicing image, including: the category of the obstacle the center point coordinates of the bounding box where the obstacle is located width height Further obtain the labeled surround view splicing image set U = u1, u2…, u i …,u I}∈R H×W×C , where u i represents the i-th surround view splicing image containing labeling information; H represents the height of the surround view splicing image, W represents the width of the surround view splicing image, and C represents the channel number of the surround view splicing image; I represents the total number of surround view splicing images; let the parking space labeling information of u i be L​i , let the obstacle annotation information of u be H i ; i ;

[0045] Step 2, as shown in Figure 3 , a deep learning-based convolutional neural network is built, which includes a backbone network module, a feature prediction module and a detection module in sequence:

[0046] Step 2.1, the backbone network module is composed of Q down-sampling blocks and P residual blocks, and the Q down-sampling blocks are DSampleBlock1,…,DSampleBlock q ,…,DSampleBlock Q ; wherein DSampleBlock q represents the qth down-sampling block, q = 1, 2,…, Q;

[0047] The qth down-sampling block DSampleBlock q is composed of the qth multi-layer two-dimensional convolutional layer Dconv2D q and the qth down-sampling layer DSample q , the qth multi-layer two-dimensional convolutional layer Dconv2D q includes X two-dimensional convolutional layers, X batch normalization layers and X LeakyReLU activation function layers, wherein the convolution kernel size of the xth two-dimensional convolutional layer is k x ; x = 1, 2,…, X;

[0048] The P residual blocks are ResBlock1,…,ResBlock p ,…,ResBlock P ; wherein ResBlock p represents the pth residual block, p = 1, 2,…, P;

[0049] The pth residual block ResBlock p is composed of r multi-layer two-dimensional convolutional layers Resconv2D, and the first multi-layer two-dimensional convolution Resconv2D P,1 in the pth residual block is connected to the rth multi-layer two-dimensional convolution Resconv2D P,r by a jump connection; increasing the residual block can enrich more shallow information and prevent overfitting.

[0050] An arbitrary annotated image u in the surround view stitching image set is input into the deep learning convolutional neural network, and after being processed by the Q down-sampling blocks and the P residual blocks in the backbone network, a feature map DResult H×W×C is obtained, wherein u ∈ R i DResult ∈ R H×W×512;

[0051] Step 2.2, the feature map DResult i In the input pyramid pooling structure, first, three maximum pooling operations are performed respectively to obtain a first pooling feature map MaxpoolDResult i,1 , a second pooling feature map MaxpoolDResult i,2 , and a third pooling feature map MaxpoolDResult i,3 . After channel splicing of the three pooling feature maps and DResult i , a pyramid pooling feature map DSPPResult i is output. H×W×C” ; C" represents the number of channels of the pyramid pooling feature map; the SPP structure can solve the problem of non-uniform input image size. Since SPP can handle different aspect ratios and different sizes of input images, the scale invariance of the image is improved and overfitting is reduced. Experiments show that the diversity of training image size is more conducive to network convergence than single-size training images.

[0052] Step 2.3, the feature prediction module is composed of a channel attention mechanism unit SE and branch 1 and branch 2 as shown in Figure 2 ;

[0053] Branch 1 is composed of k multi-layer two-dimensional convolution layers Neckconv2D1,…,Neckconv2D k , wherein the kth multi-layer two-dimensional convolution layer Neckconv2D k includes Y two-dimensional convolution layers, Y batch normalization layers, and Y LeakyReLU activation function layers, wherein the convolution kernel size of the yth two-dimensional convolution layer is k y ×k y ;

[0054] Branch 2 is composed of a residual module NeckResblock and a two-dimensional convolution layer conv2D, and the residual module NeckResblock is composed of l multi-layer two-dimensional convolution layers NeckResconv2D1,…,NeckResconv2D l , wherein the lth two-dimensional convolution layer NeckResconv2D l includes Z two-dimensional convolution layers, Z batch normalization layers, and Z LeakyReLU activation function layers, wherein the convolution kernel size of the zth two-dimensional convolution layer is k' z ×k' z ;

[0055] Pyramid pooling feature map DSPPResult iThe input feature prediction module is processed by the channel attention mechanism unit SE to obtain an average pooling feature map AavgDSPPResult with a dimension of (H, W, C") through global average pooling i The average pooling feature map AavgDSPPResult i is compressed into a feature vector with a dimension of (1, 1, C"), and then the channel dimension of the feature vector is reduced to (1, 1, C" / r) through the first fully connected layer FC1, and then processed through the Swish activation function, and then the dimension of the feature vector is raised back to (1, 1, C") through the second fully connected layer FC2, and finally converted into a normalized weight vector between 0 and 1 through the sigmoid function; the normalized weight vector and the DSPPResult i are multiplied channel by channel to generate a weighted attention mechanism feature map SEDSPPResult i ∈R H×W×C” Adding the SE module can model the correlation between different channels and automatically obtain the importance of each feature channel through network learning, and finally give different weight coefficients to each channel to strengthen important features and suppress unimportant features.

[0056] Step 2.4, the attention mechanism feature map SEDSPPResult i is input into the branch 1 and processed through convolution to obtain a feature map PslotResult containing parking space detection information i ∈R H×W×C”′ , and C'" represents the number of channels of the feature map containing parking space detection information.

[0057] The weighted feature map SEDSPPResult i is input into the branch 2 and processed through l times of convolution to obtain a feature map NeckResResult l , and then the SEDSPPResult i is concatenated with the NeckResResult l to obtain a feature map NeckConcatResult i , and then the feature map NeckConcatResult i is processed through the two-dimensional convolution layer conv2D to obtain a feature map DetectResult containing obstacle detection information

[0058] i ∈R H×W×C”” , and C'" represents the number of channels of the feature map containing parking space detection information.

[0059] ​The feature map PslotResult containing parking space detection information i In the input parking space detection module Head1, the predicted parking space information is obtained The parking space information The information of the nth directional marker arrow in the image is

[0060]

[0061] The predicted value of the origin coordinate of the nth directional marker arrow in the ith image is The predicted value of the category information of the nth directional marker arrow in the ith image is “L” when “T” when The predicted value of the cosine value and the sine value of the nth directional marker arrow in the ith image is The confidence of the nth directional marker arrow in the ith image is

[0062] The feature map DetectResult containing obstacle detection information i In the input obstacle detection module Head2, fitting is performed, and finally the predicted obstacle information is obtained The obstacle information The information of the mth obstacle bounding box is

[0063]

[0064] The predicted value of the center coordinate of the mth obstacle bounding box in the ith image is The predicted value of the width and height of the mth obstacle bounding box in the ith image is The category information of the obstacle detected by the mth bounding box in the ith image is

[0065] Step 2.5, establish a backpropagation loss function:

[0066] The feature map PslotResult containing parking space detection information i The feature map DetectResult containing obstacle detection information i are respectively divided into HxW grids, and a loss function is established according to the grids.

[0067] The first loss function Loss of PslotResult i is established by using formula (1): i1

[0068] ​

[0069] In formula (1), indicates whether there is a detection target in the hth grid in HxW grids divided in the feature map PslotResult i , if there is, let Otherwise, let indicates the confidence of the nth directional marker arrow detected in the hth grid divided in PslotResult i ; indicates the pixel coordinates of the nth directional marker arrow detected in the hth grid divided in PslotResult i ; indicates the category of the nth directional marker arrow detected in the hth grid divided in PslotResult i ; indicates the cosine value and the sine value of the nth directional marker arrow detected in the hth grid divided in PslotResult i .

[0070] The feature map DetectResult i containing obstacle detection information is detected by using the detection method in the open source algorithm YOLOv2, and 5 anchor boxes of different sizes are used in YOLOv2 to fit detection boxes of different sizes, so that the second loss function Loss i of DetectResult i2 is obtained:

[0071]

[0072] In formula (2), indicates whether there is a detection target in the nth anchor box in the uth grid in HxW grids divided in the feature map DetectResult i , if there is not, let Otherwise, let indicates the weight coefficient when there is no detection target in the uth grid; B represents the number of anchor boxes; indicates the confidence that the mth obstacle in the uth grid divided in DetectResult i is of the cth category; indicates the center pixel coordinates of the detection box of the mth obstacle detected in the uth grid divided in DetectResult i ; indicates the center pixel coordinates of the detection box of the mth obstacle detected in the uth grid divided in DetectResult i ;width and height of the detection frame of the mth obstacle detected in the uth grid divided by the middle.

[0073] The total loss function of the ith feature map with label information is established by using formula (3)

[0074]

[0075] Step 3, the set U of stitched images is input into a deep learning-based convolutional neural network for training, and the overall loss function Loss is calculated total When the number of training iterations reaches the set number of times, the training is stopped, thereby obtaining an optimal parking space and obstacle extraction network for identifying parking spaces and obstacles in automatic parking.

[0076] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the parking space and obstacle detection method, and the processor is configured to execute the program stored in the memory.

[0077] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, performs the steps of the parking space and obstacle detection method.

Claims

1. A method for detecting a parking space and an obstacle based on deep learning, characterized in that, Comprising the following steps: Step 1, the splicing of the surround view image and the preprocessing of the data set: Step 1.1, fish-eye cameras are installed at the front bumper, rear license plate, lower end of the left and right rearview mirrors of the vehicle to obtain images collected by each fish-eye camera, and the images are respectively corrected for distortion, perspective transformation and spliced into a surround view image, and the surround view image is optimized in the manner of feathering the overlapping area to obtain an optimized surround view splicing image; Step 1.2, surround view splicing images of the vehicle driving near a parking space are obtained in the manner of step 1.1, and the parking space in the i-th surround view splicing image is labeled, if the parking space in the i-th surround view splicing image is at the edge, the edge parking space corner point is taken as the origin, and two directional ray arrows are drawn to form a "L"-shaped directional marker arrow of the parking space; If the parking space is not at the edge, the common corner point of the parking space is taken as the origin, and three directional ray arrows are drawn to form a "T"-shaped directional marker arrow; Thus, the i-th loop view spliced image n-th parking space marking information is obtained, including: origin coordinates The shape type of the directional marker arrow And the rotation angle of the directional marker arrow Obstacles in the i-th surround view stitched image are also labeled, and the labeling information of the m-th obstacle in the i-th surround view stitched image includes: the category of the obstacle. Coordinates of the center point of the bounding box containing the obstacle width high This results in a set of annotated surround view stitched images U = {u1, u2, ..., u...} including parking space and obstacle marking information. i ..., u I }∈R H×W×C , where u i Let u represent the i-th surround view stitched image containing annotation information; H represents the height of the surround view stitched image, W represents the width of the surround view stitched image, C represents the number of channels of the surround view stitched image; I represents the total number of surround view stitched images; let u i The parking space marking information is recorded as L i , let u i The obstacle labeling information is denoted as H. i ; Step 2, a convolutional neural network based on deep learning is built, which sequentially includes a backbone network module, a pyramid pooling structure, a feature prediction module, a parking space detection module Head1, and an obstacle detection module Head2: Step 2.1, the backbone network module is composed of Q down-sampling blocks and P residual blocks, the Q down-sampling blocks are DSampleBlock1,..., DSampleBlock q ,..., DSampleBlock Q ; wherein DSampleBlock q denotes the qth down-sampling block, q = 1, 2,..., Q; Step 2.2, the residual block is composed of a plurality of convolutional layers, and the residual block is denoted as ResidualBlock1, ResidualBlock2,..., ResidualBlockP; wherein ResidualBlockP denotes the Pth residual block, P = 1, 2,..., P; The qth level downsampling block DSampleBlock q by the qth level multi-layer two-dimensional convolution layer Dconv2D q and the qth level downsampling layer DSample q ; wherein the qth level multi-layer two-dimensional convolution layer Dconv2D q includes X two-dimensional convolution layers, X batch normalization layers and X LeakyReLU activation function layers, wherein the convolution kernel size of the xth two-dimensional convolution layer is k x ×k x ; The P residual blocks are ResBlock1, ..., ResBlock2. p , ..., ResBlock P Among them, ResBlock p This represents the p-th level residual block, where p = 1, 2, ..., P; the pth residual block ResBlock p consists of r multi-layer two-dimensional convolution layers Resconv2D, and the pth residual block ResBlock p the first multi-layer two-dimensional convolution Resconv2D P，1 is connected by a skip connection with the rth multi-layer two-dimensional convolution Resconv2D P，r ​ Any one of the labeled images u in the set of labeled surround view stitching images U is input into the deep learning convolutional neural network i Input into the deep learning convolutional neural network and sequentially processed by the Q down-sampling blocks and the P residual blocks of the backbone network to obtain a feature map DResult i ∈R H×W×C′ ; C' represents the channel number of the feature map DResult processed by the backbone network i ​ Step 2.2, the feature map DResult i In the pyramid pooling structure, first, three maximum pooling operations are performed respectively to obtain a first pooling feature map MaxpoolDResult i，1 , a second pooling feature map MaxpoolDResult i，2 , and a third pooling feature map MaxpoolDResult i，3 After channel splicing of the three pooling feature maps and DResult i , a pyramid pooling feature map DSPPResult i ∈RH ×W×c″ is output; C" represents the number of channels of the pyramid pooling feature map. Step 2.3, the feature prediction module is composed of a channel attention mechanism unit SE and branches 1 and 2; The branch 1 is composed of k multi-layer two-dimensional convolution layers Neckconv2D1,..., Neckconv2D k , wherein the kth multi-layer two-dimensional convolution layer Neckconv2D k includes Y two-dimensional convolution layers, Y batch normalization layers and Y LeakyReLU activation function layers, wherein the convolution kernel size of the yth two-dimensional convolution layer is k y ×k y ; The branch 2 is composed of a residual module NeckResblock and a two-dimensional convolution layer conv2D, and the residual module NeckResblock is composed of l multi-layer two-dimensional convolution layers NeckResconv2D1,..., NeckResconv2D l l, wherein the lth two-dimensional convolution layer NeckResconv2D l includes Z two-dimensional convolution layers, Z batch normalization layers and Z LeakyReLU activation function layers, wherein the convolution kernel size of the zth two-dimensional convolution layer is k' z ×k' z ; the pyramid-pooled feature map DSPPResult i The input feature prediction module is processed by the channel attention mechanism unit SE first for global average pooling to obtain an average-pooled feature map AavgDSPPResult with dimensions (H, W, C″) i The average-pooled feature map AavgDSPPResult i is compressed into a feature vector with dimensions (1, 1, C″), and then the channel dimension of the feature vector is reduced to (1, 1, C″ / r) by the first fully connected layer FC1, and then processed by the Swish activation function, and then the dimension of the feature vector is raised back to (1, 1, C″) by the second fully connected layer FC2, and finally converted into a normalized weight vector between 0 and 1 by the sigmoid function; the normalized weight vector and the DSPPResult i After channel-by-channel multiplication, a weighted attention mechanism feature map SEDSPPResult is generated i ∈R H×W×C" ; Step 2.4, the attention mechanism feature map SEDSPPResult i The input branch 1 is subjected to convolution processing to obtain a feature map PslotResult containing parking space detection information i ∈R H×W×C″′ C"'represents the number of channels of the feature map containing parking space detection information The weighted feature map SEDSPPResult i In the input branch 2, and after l times of convolution processing, the feature map NeckResResult l , SEDSPPResult i is concatenated with NeckResResult l to obtain the feature map NeckConcatResult i , and then the feature map NeckConcatResult i is processed by the two-dimensional convolution layer conv2D to obtain the feature map DetectResult containing obstacle detection information i ∈R H×W×C , and C'" represents the channel number of the feature map containing obstacle detection information. The feature map PslotResult containing parking space detection information i In the input parking space detection module Head1, the predicted parking space information is obtained Wherein, the parking space information The information of the nth directional marker arrow in the jth image is is the predicted value of the origin coordinates of the nth directional marker arrow in the ith image, is the predicted value of the category information of the nth directional marker arrow in the ith image, when is "L" type, when is "T" type; respectively, are the predicted values of the cosine value and the sine value of the nth directional marker arrow in the ith image, is the confidence of the nth directional marker arrow in the ith image; The feature map DetectResult containing the obstacle detection information i The fitting is performed in the input obstacle detection module Head2, and finally the predicted obstacle information is obtained The obstacle information The information of the mth obstacle bounding box in the i th picture is is the predicted value of the center coordinate of the mth obstacle bounding box in the i th picture, is the predicted value of the width and height of the mth obstacle bounding box in the i th picture, is the category information of the obstacle detected by the mth bounding box in the i th picture; Step 2.5, a back propagation loss function is established: Information on directional marker arrows A first loss function Loss1 is calculated by performing a mean square error loss calculation. Positioning information of an obstacle detection frame The mean square error loss is calculated, and The binary cross entropy loss is calculated to obtain a second loss function Loss2; thereby calculating a total loss function Loss total = Loss1 + Loss2; Step 3, input the ring view spliced image set U into a deep learning based convolutional neural network for training, and calculate the overall loss function Loss total When the number of training iterations reaches the set number of times, the training is stopped, thereby obtaining an optimal parking space and obstacle extraction network for identifying the parking space and the obstacle in automatic parking.

2. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the parking space and obstacle detection method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is run by the processor to execute the steps of the parking space and obstacle detection method of claim 1.

Images