Autonomous driving vehicle parking space detection method based on panoramic around view image

Through the panoramic surround-view image detection method, four cameras and deep learning networks are used to solve the problem of high cost and poor accuracy of parking space detection in driverless parking, and low-cost and high-precision parking space detection is achieved, suitable for multiple A parking environment.

WO2024229994A1PCT designated stage expired Publication Date: 2024-11-14EAST CHINA JIAOTONG UNIVERSITY +1

Patent Information

Application Number
PCT/CN2023/111468
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-09
Filing Date
2023-08-07
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

The prior art has high cost in parking space detection in driverless parking environments, and the image fusion is ineffective, and the detection accuracy is poor.

Method used

Adopt unmanned parking space detection method based on panoramic circumferential viewing images, and obtain environmental images through four cameras, perform dedistortion and perspective transformation, use RANSAC algorithm to extract feature points, and combine with Laplace fusion algorithm to generate panoramic circumferential view. And use the deep learning Rep-VGG network for parking space detection.

Benefits of technology

Reduces costs, improves image fusion quality and detection accuracy, achieves a more natural stitching transition and a more complete surround image display, with higher robustness and real-time performance, and is suitable for a variety of parking environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023111468_14112024_PF_FP_ABST
    Figure CN2023111468_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present invention is an autonomous driving vehicle parking space detection method based on a panoramic around view image. The present invention uses a Laplacian image blending method for processing the splicing of a panoramic around view image, so as to achieve effective blending of images, and, according to an image evaluation coefficient, uses an adjustment function so as to ensure the splicing quality while reducing the algorithm complexity; furthermore, the transition of splicing positions of the generated panoramic around view image in the present invention is more natural, so that the around view image can be displayed more completely and attractively. Compared with conventional machine vision detection techniques, the parking space detection method provided by the present invention uses a deep learning-based object detection technique to detect parking space lines and therefore has higher detection precision and higher robustness, and uses a relatively light-weight network design to ensure the real-time performance of the detection algorithm.
Need to check novelty before this filing date? Find Prior Art

Description

Parking space detection method for unmanned vehicle based on panoramic surround view images Technical Field

[0001] The present invention relates to the field of intelligent automobile technology, and in particular to a method for detecting parking spaces for an unmanned vehicle based on panoramic surround view images. Background Art

[0002] With the development of automobile technology, more and more smart cars are equipped with advanced driver assistance systems. Perception technology, as one of the core technologies of assisted driving, has now become an important channel for smart cars to transition from assisted driving to unmanned driving.

[0003] The increasing number of vehicles in cities is posing challenges to urban transportation. Urban parking spaces are struggling to keep pace with the growth in vehicles, leading to increasingly severe and complex urban traffic conditions and exacerbating the shortage of parking resources. Parking space detection technology is a key component of autonomous parking environment perception systems. The effectiveness of parking space detection significantly impacts the safety and stability of the entire parking process. This technology allows autonomous vehicles to obtain real-time information about surrounding parking spaces based on parking space markings on the ground and accurately determine whether a parking space is available.

[0004] Existing technologies mainly obtain information about the vehicle's surrounding environment through a certain number of cameras and obtain parking space information based on machine vision detection technology. However, the method of fusing multiple cameras increases costs, and existing technologies cannot effectively fuse images, and the algorithm's detection accuracy is also poor.

[0005] Summary of the Invention

[0006] The purpose of the present invention is to provide a method for detecting parking spaces for unmanned parking vehicles based on panoramic surround view images, so as to solve the technical problems of the existing technology such as high cost, ineffective image fusion and poor detection accuracy.

[0007] A method for detecting parking spaces for an unmanned vehicle based on panoramic surround view images comprises the following steps:

[0008] Step 1: Obtain a front-view camera original image, a rear-view camera original image, a left-view camera original image, and a right-view camera original image of the vehicle's surrounding environment through a front-view camera, a rear-view camera, a left-view camera original image, and a right-view camera original image, respectively, provided on the vehicle;

[0009] Step 2: Dedistort and perspective transform the original image of the front camera, the original image of the rear camera, the original image of the left camera, and the original image of the right camera, respectively, to obtain a front view, a rear view, a left view, and a right view;

[0010] Step 3: Use the RANSAC algorithm to extract feature points from the front view, back view, left view, and right view to determine the area to be spliced ​​and fused;

[0011] Step 4: The areas to be stitched and fused are fused using the Laplace fusion algorithm to obtain a panoramic ring view. During the fusion process, the number of layers of the Laplace fusion algorithm is adjusted according to the image quality evaluation function.

[0012] Step 5: Input the panoramic surround view into the trained deep learning-based parking space detection model to obtain parking space attribute results and parking space identification point results. The parking space detection model uses the Rep-VGG network as the network backbone. After the network backbone output passes through the Sim-CSPSPPF module and several residual convolution modules, the branches of the backbone network will also pass through several residual convolution modules. Finally, different parking space results are output on feature maps of different resolutions. Among them, the parking space attribute results are predicted on the feature map with lower resolution, and the parking space identification point results are predicted on the feature map with higher resolution.

[0013] The method for detecting parking spaces for unmanned vehicles based on panoramic surround view images provided by the present invention has the following beneficial effects:

[0014] (1) The parking space detection method provided by the present invention only requires four cameras to complete tasks such as image acquisition and stitching, which is more cost-effective;

[0015] (2) The present invention adopts the Laplace image fusion method to process the stitching of the panoramic view, which can effectively fuse the images. According to the image evaluation coefficient, an adjustment function is used to ensure the stitching quality while reducing the algorithm complexity. The transition of the stitching of the panoramic view generated by the present invention is more natural, and the panoramic image display is more complete and beautiful.

[0016] (3) Compared with traditional machine vision detection technology, the parking space detection method provided by the present invention uses deep learning-based target detection technology to detect parking space lines, which has higher detection accuracy and stronger robustness. It adopts a relatively lightweight network design to ensure the real-time performance of the detection algorithm.

[0017] (4) The parking space detection method provided by the present invention does not depend on the relative spatial position of vehicles near the target parking space. The parking space detection task can be completed as long as there are parking markings that meet parking standards on the ground of the parking area;

[0018] (5) The parking space detection method provided by the present invention is applicable to a variety of parking environments, and can detect vertical, parallel, and inclined parking spaces and obtain information on whether the parking space is occupied. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 is a flow chart of a method for detecting parking spaces for an unmanned vehicle based on panoramic surround view images according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of the installation position of the camera. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0022] Referring to FIG. 1 , an embodiment of the present invention provides a method for detecting parking spaces for an unmanned vehicle based on panoramic surround view images, comprising the following steps:

[0023] Step 1: Obtain a front view camera original image, a rear view camera original image, a left view camera original image, and a right view camera original image of the vehicle's surrounding environment through a front view camera, a rear view camera, a left view camera original image, and a right view camera original image respectively set on the vehicle.

[0024] In this embodiment, the installation positions of the front-view camera, rear-view camera, left-view camera and right-view camera are shown in FIG2 . These four cameras are all fisheye cameras with a viewing angle of 190°, so that the perception range can completely cover the vehicle's surrounding environment.

[0025] Step 2: Dedistort and perspective transform the original images of the front camera, rear camera, left camera, and right camera to obtain the front view, rear view, left view, and right view, respectively.

[0026] In this embodiment, only the radial distortion and tangential distortion of the fisheye camera are considered.

[0027] Step 2 specifically includes:

[0028] S201, converting the physical coordinates in the world coordinate system into pixel coordinates in the pixel coordinate system using the following formula:

[0029] Among them, Z is the scale factor, D is the distortion matrix, R is the rotation matrix, T is the translation vector, and A is the internal parameter matrix. E is the external parameter matrix, (U, V, W) is the physical coordinate of the target point in the world coordinate system, (u, v) is the pixel coordinate of the target point in the ideal undistorted pixel coordinate system, and fx 、f y are the normalized focal lengths on the x-axis and y-axis, respectively, and f x =f / dx,f y =f / dy, f represents the focal length of the camera, dx and dy are pixel units, (C x , C y ) is the image origin;

[0030] S202, fix the world coordinate system on the chessboard, then the physical coordinates W of any point on the chessboard are 0, and obtain a number of (u, v), (U, V) through calibration to obtain the parameters of the intrinsic parameter matrix A;

[0031] S203, based on the radial distortion formula and the tangential distortion formula, obtain the matrix form of the ideal conversion relationship between image coordinates and pixel coordinates:

[0032] The radial distortion formula is:

[0033] The tangential distortion formula is:

[0034] Among them, D1 and D2 are the intermediate values ​​of the distortion matrix, (x, y), are the ideal undistorted normalized image coordinates and the distorted normalized image coordinates respectively; r is the distance from the image pixel to the image center point, k1, k2, k3, p1, and p2 are distortion parameters, (u, v) and (u^, v) are the ideal undistorted pixel coordinates and the distorted pixel coordinates, respectively. The conversion relationship between the ideal image coordinates and pixel coordinates is:

[0035] S204, obtaining distortion parameters D1 and D2 through multiple offline calibration images, thereby determining a distortion matrix D, and dedistorting the front camera original image, the rear camera original image, the left camera original image, and the right camera original image according to the distortion matrix D, to obtain a dedistorted front camera original image, a dedistorted rear camera original image, a dedistorted left camera original image, and a dedistorted right camera original image;

[0036] S205. Based on the extrinsic parameter matrix E obtained by extrinsic calibration, the dedistorted original image of the front view camera, the dedistorted original image of the rear view camera, the dedistorted original image of the left view camera, and the dedistorted original image of the right view camera are respectively subjected to perspective transformation processing to obtain a front view, a rear view, a left view, and a right view.

[0037] As shown in FIG2 , a front view A, a right view B, a rear view C, and a left view D are obtained respectively.

[0038] Step 3: Use the RANSAC algorithm to extract feature points from the front view, back view, left view, and right view to determine the area to be spliced ​​and fused.

[0039] As shown in FIG2 , a feature-based image stitching method is used for the area requiring image registration, and the RANSAC algorithm is used to extract feature points to determine the areas AB, BA, BC, CB, CD, DC, DA, and AD requiring stitching and fusion.

[0040] Step 4: The areas to be stitched and fused are fused using the Laplace fusion algorithm to obtain a panoramic ring view. During the fusion process, the number of layers of the Laplace fusion algorithm is adjusted according to the image quality evaluation function.

[0041] In step 4, the Laplace fusion algorithm is used to fuse the areas to be spliced ​​and fused, specifically including:

[0042] S401a, for one of the two images to be stitched and fused, first perform Gaussian smoothing and downsampling, then perform Gaussian smoothing and upsampling, repeat N times, and obtain the N+1-layer Gaussian pyramid of image P1. When performing the i-th Gaussian smoothing and downsampling, the first intermediate image P_down of the i-th layer is obtained. i , when performing the i-th Gaussian smoothing and upsampling, the second intermediate image P_up of the i-th layer is obtained i , i≤N, the N+1-layer Gaussian pyramid of image P1 consists of N second intermediate images P_up i and image P1;

[0043] S402a, construct the Laplace pyramid using the following formula:

[0044] Among them, P1_L i Represents the Laplacian residual image of the i-th layer of image P1, UP is the upsampling operation, N Laplacian residual images P1_L i Constructing the N-layer Laplacian pyramid, by the Laplacian residual image P1_L of the N-th layer N Perform Gaussian downsampling to obtain the third intermediate image P1_L of image P1 top ;

[0045] S403a, according to the method of steps S401a to S402a, the other image P2 of the two images to be spliced ​​and fused is processed accordingly to obtain the Laplacian residual image P2_L of the i-th layer of the image P2 i, thereby obtaining the third intermediate image P1_L containing the image P1 top The N+1 layer P1 image Laplacian pyramid and the third intermediate image P2_L containing the image P2 top The N+1 layer P2 image Laplacian pyramid;

[0046] S404a, set a binary mask value mask matrix, the mask matrix represents the weight ratio required for the fusion of image P1 and image P2, and downsample the mask matrix N times to obtain the downsampled matrix mask_down i , thereby constructing the N+1 layer Gaussian pyramid of the mask matrix, according to the N+1 layer Gaussian pyramid of the mask matrix, P1_L i With P2_L i Perform image mask fusion to obtain Represents the mask fusion image of the i-th layer image, and its expression is:

[0047] Among them, * is the image layer fusion operator;

[0048] S405a, use the following formula Reconstruct and get the i-th layer reconstructed fusion effect image

[0049] Among them, UP is the upsampling operation, represents the i+1th layer image mask fusion image, represents the i-1th layer reconstructed fusion effect image, * is the image layer fusion operator;

[0050] S406a, starting from i=0, repeat S405a until i=N, to construct the Nth layer of reconstructed fusion effect image Will As the final result of image fusion.

[0051] In step 4, adjusting the number of layers of the Laplace fusion algorithm according to the image quality evaluation function specifically includes:

[0052] S401b, the image stitching quality of the image fusion area is evaluated. The SSIM evaluation method is used to evaluate the image quality from three aspects: brightness, contrast, and structural similarity. The expression of the SSIM evaluation function is:

[0053] Among them, S(Iori , I fuse ) is the evaluation coefficient, S(I ori , I fuse )∈(0,1),l α (I ori , I fuse ), c β (I ori , I fuse ),s γ (I ori , I fuse ) represent brightness similarity, contrast similarity, and structural similarity respectively, Iori and Ifuse represent the original image and the image to be evaluated respectively, α, β, and γ are the weighted contribution values ​​of brightness similarity, contrast similarity, and structural similarity to the score respectively; μ o and μ f are the average intensities of the original image and the image to be evaluated, σ o and σ f Represents the pixel standard deviation of the original image and the image to be evaluated, σ o-f Represents the mutual correlation coefficient between the original image and the image to be evaluated, C1, C2, and C3 are initial constants;

[0054] S402b, based on the evaluation coefficient, according to the adjustment function A(N, S), adjust the number of layers N of the Laplace fusion algorithm. The expression of the adjustment function A(N, S) is:

[0055] Where e is a constant, S is an adjustable parameter, S∈(0,1).

[0056] The regions to be fused, AB and BA, BC and CB, CD and DC, and DA and AD, are respectively used as the P1 and P2 images of the Laplace algorithm. By performing the above operations, four smooth fused images (upper left, upper right, lower right, and lower left) can be obtained. By splicing them with other regions that do not need to be fused, a complete, smooth, and clear panoramic ring view can be obtained.

[0057] Step 5: Input the panoramic surround view into the trained deep learning-based parking space detection model to obtain parking space attribute results and parking space identification point results. The parking space detection model uses the Rep-VGG network as the network backbone. After the network backbone output passes through the Sim-CSPSPPF module and several residual convolution modules, the branches of the backbone network will also pass through several residual convolution modules. Finally, different parking space results are output on feature maps of different resolutions. Among them, the parking space attribute results are predicted on the feature map with lower resolution, and the parking space identification point results are predicted on the feature map with higher resolution.

[0058] Among them, step 5 specifically includes:

[0059] S501, design the expression of the total loss function Loss of the parking space detection model based on deep learning: Loss = λ PKL ·L PKL +λ MP ·L MP

[0060] Among them, λ PKL is the parking space loss function L PKL The weight hyperparameter, λ MP is the loss function L of the parking sign point MP The weight hyperparameters are used to design the parking loss function L on the feature map with lower resolution. PKL , on the feature map with higher resolution, design the loss function L of the parking sign point MP .

[0061] S502, setting the initial learning rate to 0.001, selecting AdamW as the gradient descent optimizer, setting the batch size to 32, and the initial number of iterations to 200. When the value of the total loss function Loss converges, a trained parking space detection model based on deep learning is obtained;

[0062] At step S503, the panoramic surround view is input into the trained deep learning-based parking space detection model to obtain information about two parking space identification points on the parking space entrance line. The parking space attribute results and the parking space identification point results are combined to obtain the coordinate information M1 and M2 of the two identification points on the target parking space entrance line. The coordinate information M3 and M4 of the remaining two identification points of the target parking space that do not appear in the panoramic surround view is obtained using the parking space identification point derivation expression. The parking space identification point derivation expression is:

[0063] Among them, α n Indicates the angle of the nth parking space in the panoramic view, L n Indicates the depth of the nth parking space in the panoramic surround view.

[0064] Parking space loss function L PKL The expression is:

[0065] Among them, W T Represents the parking space category loss function L T The weight hyperparameter, W O Denotes the parking space occupancy loss function L O The corresponding weight hyperparameter, W C Represents the parking position confidence loss function LC The corresponding weight hyperparameters;

[0066] Parking space category loss function L T The expression is:

[0067] Among them, w1 represents the width of the feature map with lower resolution, h1 represents the height of the feature map with lower resolution, Indicates whether the center of the g-th grid belongs to the parking area, Indicates that the center of the g-th grid belongs to the parking area, Indicates that the center of the g-th grid does not belong to the parking area; x t represents the probability that the predicted parking space belongs to the tth type of parking space; y t Indicates the true value of the predicted parking space belonging to the t-th parking space category, y t =(0,0,1) indicates a vertical parking space, y t =(0, 1, 0) indicates a parallel parking space, y t =(1,0,0) represents an inclined parking space, σ(x t ) is an intermediate function;

[0068] Parking space occupancy loss function L O The expression is:

[0069] Among them, α is the weight value for balancing effective parking spaces and invalid parking spaces, Indicates whether the g-th grid belongs to the valid parking space area, Indicates that the center point of the g-th grid is within the valid parking space area. Indicates that the center point of the g-th grid is located in the non-valid parking space area; Indicates whether the g-th grid belongs to the invalid parking space area, Indicates that the center point of the g-th grid is located in the invalid parking space area. Indicates that the center point of the g-th grid is located in the non-invalid parking space area; Indicates the probability of predicting whether the parking space on the g-th grid is valid, Indicates the true value of whether the predicted parking space on the g-th grid is valid, Indicates a valid parking space. Indicates an invalid parking space;

[0070] Parking location confidence loss function L C The expression is:

[0071] in, Indicates whether the g-th grid belongs to the parking area, Indicates that the center of the g-th grid belongs to the parking area, Indicates that the center of the g-th grid does not belong to the parking area; represents the predicted probability value that the g-th grid belongs to a parking space; Indicates the true value that the g-th grid belongs to the parking space; α P The weight value for balancing parking areas and non-parking areas.

[0072] The loss function L of the parking sign MP The expression is:

[0073] in, Loss function representing the coordinates of parking space identification points The corresponding weight hyperparameter, W θ The loss function L representing the angular orientation of the parking space marking point θ The corresponding weight hyperparameters;

[0074] Loss function of parking space identification point coordinates The expression is:

[0075] Among them, w2 represents the width of the feature map with higher resolution, h2 represents the height of the feature map with higher resolution, Indicates whether the g-th grid contains a parking space identification point, Indicates that the g-th grid contains the parking space identification point, Indicates that the g-th grid does not contain a parking space identification point; Represents the predicted value of the x-coordinate direction of the parking space identification point on the g-th grid, Represents the true value of the x-coordinate direction of the parking space identification point on the g-th grid; Represents the predicted value of the y-coordinate direction of the parking space identification point on the g-th grid, Represents the true value of the y-coordinate direction of the parking space identification point on the g-th grid;

[0076] The loss function L of the parking space marking point angle orientation θ The expression is:

[0077] in, Indicates the cosine value of the angle orientation of the predicted parking space identification point on the g-th grid, Indicates the true value of the cosine value of the parking space identification point on the g-th grid; Represents the sin value of the angle of the predicted parking space identification point on the g-th grid, Indicates the true value of the sin value of the angle of the parking space identification point on the g-th grid.

[0078] Finally, by combining the information of the four parking space identification points in the panoramic surround view and the parking space attribute information, all detailed information of the parking spaces around the vehicle can be obtained in real time. The detailed information of the parking spaces can be transmitted to the next level of autonomous driving path planning and decision control module to realize the automatic parking function.

[0079] In summary, the method for detecting parking spaces for autonomous vehicles based on panoramic surround view images provided by this embodiment has the following beneficial effects:

[0080] (1) The parking space detection method provided by the present invention only requires four cameras to complete tasks such as image acquisition and stitching, which is more cost-effective;

[0081] (2) The present invention adopts the Laplace image fusion method to process the stitching of the panoramic view, which can effectively fuse the images. According to the image evaluation coefficient, an adjustment function is used to ensure the stitching quality while reducing the algorithm complexity. The transition of the stitching of the panoramic view generated by the present invention is more natural, and the panoramic image display is more complete and beautiful.

[0082] (3) Compared with traditional machine vision detection technology, the parking space detection method provided by the present invention uses deep learning-based target detection technology to detect parking space lines, which has higher detection accuracy and stronger robustness. It adopts a relatively lightweight network design to ensure the real-time performance of the detection algorithm.

[0083] (4) The parking space detection method provided by the present invention does not depend on the relative spatial position of vehicles near the target parking space. The parking space detection task can be completed as long as there are parking markings that meet parking standards on the ground of the parking area;

[0084] (5) The parking space detection method provided by the present invention is applicable to a variety of parking environments, and can detect vertical, parallel, and inclined parking spaces and obtain information on whether the parking space is occupied.

[0085] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0086] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A parking space detection method for unmanned driving based on panoramic surround image, It is characterized in that The following steps are involved: Step 1, obtaining a front view camera original image, a rear view camera original image, a left view camera original image, and a right view camera original image of the surrounding environment of the vehicle respectively through a front view camera, a rear view camera, a left view camera, and a right view camera arranged on the vehicle; Step 2, respectively performing dedistortion processing and perspective transformation processing on the original image of the front camera, the original image of the rear camera, the original image of the left camera, and the original image of the right camera to obtain a front view, a rear view, a left view, and a right view; Step 3, using the RANSAC algorithm to extract feature points from the front view, back view, left view, and right view to determine the area to be spliced ​​and fused; Step 4, the areas to be stitched and fused are fused by Laplace fusion algorithm to obtain a panoramic ring view, and in the fusion process, the number of layers of Laplace fusion algorithm is adjusted according to the image quality evaluation function; Step 5: Input the panoramic surround view into the trained parking space detection model based on deep learning to obtain the parking space attribute results and the parking space identification point results. The parking space detection model uses the Rep-VGG network as the network backbone. After the network backbone output passes through the Sim-CSPSPPF module and several residual convolution modules, the branches of the backbone network will also pass through several residual convolution modules, and finally output different parking space results on feature maps of different resolutions, where the parking space attribute results are predicted on the feature map with lower resolution, and the parking space identification point results are predicted on the feature map with higher resolution.

2. The method for detecting parking spaces for unmanned vehicles based on panoramic surround images according to claim 1, It is characterized in that Step 2 specifically includes: S201, converting the physical coordinates in the world coordinate system into pixel coordinates in the pixel coordinate system by the following formula: Among them, Z is the scale factor, D is the distortion matrix, R is the rotation matrix, T is the translation vector, and T is the internal parameter matrix, E is the external parameter matrix, (U, V, W) is the physical coordinate of the target point in the world coordinate system, (u, v) is the pixel coordinate of the target point in the ideal undistorted pixel coordinate system, and f x 、f y are the normalized focal lengths on the x-axis and y-axis respectively, and f x =f / dx,f y =f / dy, f represents the focal length of the camera, dx and dy are pixel units, (C x , C y ) is the image origin; S202, fix the world coordinate system on the chessboard, then the physical coordinates W of any point on the chessboard are 0, and obtain a number of (u, v), (U, V) through calibration to obtain the parameters of the internal parameter matrix A; S203, based on the radial distortion formula and the tangential distortion formula, a matrix form of the conversion relationship between the ideal image coordinates and the pixel coordinates is obtained: The radial distortion formula is: The tangential distortion formula is: Among them, D 1 , D 2 is the middle value of the distortion matrix, (x,y), are the ideal undistorted normalized image coordinates and the distorted normalized image coordinates; r is the distance from the image pixel to the image center point, k 1 , k 2 , k 3 、p 1 、p 2 is the distortion parameter, (u, v), They are the ideal undistorted pixel coordinates and the distorted pixel coordinates respectively. The conversion relationship between the ideal image coordinates and the pixel coordinates is: S204, obtaining distortion parameters through multiple offline calibration images to determine D 1 , D 2 , thereby determining the distortion matrix D, and dedistorting the original image of the front camera, the original image of the rear camera, the original image of the left camera, and the original image of the right camera according to the distortion matrix D, to obtain the dedistorted original image of the front camera, the dedistorted original image of the rear camera, the dedistorted original image of the left camera, and the dedistorted original image of the right camera; S205, according to the extrinsic parameter matrix E obtained by the extrinsic parameter calibration, the dedistorted original image of the front view camera, the dedistorted original image of the rear view camera, the dedistorted original image of the left view camera, and the dedistorted original image of the right view camera are respectively processed by perspective transformation to obtain a front view, a rear view, a left view, and a right view.

3. The method for detecting parking spaces for unmanned vehicles based on panoramic surround images according to claim 1, It is characterized in that In step 4, the Laplace fusion algorithm is used to fuse the areas to be spliced ​​and fused, specifically including: S401a, for one of the two images to be stitched and fused, first perform Gaussian smoothing and downsampling, then perform Gaussian smoothing and upsampling, repeat N times, and obtain the N+1 layers of Gaussian pyramid of image P1. When performing the i-th Gaussian smoothing and downsampling, obtain the first intermediate image P_down with the i-th layer. i , when performing the i-th Gaussian smoothing and upsampling, the second intermediate image P_up of the i-th layer is obtained i , i≤N, the N+1-layer Gaussian pyramid of image P1 consists of N second intermediate images P_up i and image P1; S402a, construct the Laplace pyramid using the following formula: P1_L i =P_down i -UP(P_down i+1 ) Among them, P1_L i represents the Laplacian residual image of the i-th layer of image P1, UP is the upsampling operation, and N Laplacian residual images P1_L i Constructing an N-layer Laplacian pyramid, by performing the Laplacian residual image P1_L of the N-th layer N Perform Gaussian downsampling to obtain the third intermediate image P1_L of image P1 top ; S403a, according to the method of steps S401a to S402a, the other image P2 of the two images to be spliced ​​and fused is processed accordingly to obtain the Laplacian residual image P2_L of the i-th layer of the image P2 i , thereby obtaining a third intermediate image P1_L containing the image P1 top The N+1 layer P1 image Laplacian pyramid and the third intermediate image P2_L containing the image P2 top The N+1-layer P2 image Laplacian pyramid; S404a, set a binary mask value mask matrix, the mask matrix represents the weight ratio of the image P1 and the image P2 required for fusion, and downsample the mask matrix N times to obtain a downsampled matrix mask_down i , thereby constructing the N+1-layer Gaussian pyramid of the mask matrix. According to the N+1-layer Gaussian pyramid of the mask matrix, P1_L i With P2_L i Perform image mask fusion to obtain It represents the fusion image of the i-th layer image mask, and its expression is: Among them, * is the image layer fusion operator; S405a, use the following formula Reconstruct and get the i-th layer reconstructed fusion effect image Among them, UP is the upsampling operation, represents the i+1th layer image mask fusion image, represents the i-1th layer reconstructed fusion effect image, * is the image layer fusion operator; S406a, starting from i=0, repeat S405a until i=N, constructing the Nth layer of reconstructed fusion effect image Will As the final result of image fusion.

4. The method for detecting parking spaces for unmanned vehicles based on panoramic surround images according to claim 3, It is characterized in that In step 4, according to the image quality evaluation function, the number of layers of the Laplace fusion algorithm is adjusted, specifically including: S401b, the image stitching quality of the image fusion area is evaluated, and the SSIM evaluation method is used to evaluate the image quality from three aspects: brightness, contrast and structural similarity. The expression of the SSIM evaluation function is: S(I ori ,I fuse )=(l α (I ori ,I fuse ),c β (I ori ,I fuse ),s γ (I ori ,I fuse )) Among them, S(I ori , I fuse ) is the evaluation coefficient, S(I ori , I fuse )∈(0,1),l α (I ori , I fuse ),c β (I ori , I fuse ),s γ (I ori , I fuse ) represent brightness similarity, contrast similarity, and structure similarity respectively, I ori ,I fuse Represent the original image and the image to be evaluated respectively, α, β, γ are the weighted contribution values ​​of brightness similarity, contrast similarity, and structure similarity to the score respectively; μ o and μ f are the average intensity of the original image and the image to be evaluated, σ o and σ f Represent the pixel standard deviation of the original image and the image to be evaluated, σ o-f Represents the mutual correlation coefficient between the original image and the image to be evaluated, C 1 , C 2 , C 3 is the initial constant; S402b, based on the evaluation coefficient, according to the adjustment function A(N, S), the number of layers N of the Laplace fusion algorithm is adjusted. The expression of the adjustment function A(N, S) is: A(N,S)=20e(S-1) 4 +10(S-1) 2 +1 Among them, e is a constant, S is an adjustable parameter, S∈(0,1).

5. The method for detecting parking spaces for unmanned vehicles based on panoramic surround images according to claim 1, It is characterized in that Step 5 specifically includes: S501, design the expression of the total loss function Loss of the parking space detection model based on deep learning as follows: Loss=λ PKL ·L PKL +λ MP ·L MP Among them, λ PKL is the parking space loss function L PKL The weight hyperparameter, λ MP For parking signs Loss function L MP The weight hyperparameters are used to design the parking space loss function L on the feature map with lower resolution. PKL , on the feature map with higher resolution, design the loss function L of the parking sign point MP ; S502, setting the initial learning rate to 0.001, selecting AdamW as the gradient descent optimizer, setting the batch size to 32, and the number of iterations to 200. When the value of the total loss function Loss converges, a trained parking space detection model based on deep learning is obtained; S503, input the panoramic view into the trained parking space detection model based on deep learning, obtain the information of the two parking space identification points of the parking space entrance line, combine the parking space attribute results and the parking space identification point results to obtain the coordinate information M1 and M2 of the two identification points of the target parking space entrance line, and obtain the coordinate information M3 and M4 of the other two identification points of the target parking space that do not appear in the panoramic view through the parking space identification point derivation expression. The parking space identification point derivation expression is: Among them, α n Indicates the angle of the nth parking space in the panoramic view, L n Indicates the depth of the nth parking space in the panoramic surround view.

6. The method for detecting parking spaces for unmanned vehicles based on panoramic surround images according to claim 5, It is characterized in that Parking space loss function L PKL The expression is: L PKL =W T ·L T +W O ·L O +W C ·L C Among them, W T Represents the parking space category loss function L T The weight hyperparameter, W O Denotes the parking space occupancy loss function L O The corresponding weight hyperparameter, W C Represents the parking position confidence loss function L C The corresponding weight hyperparameters; Parking space category loss function L T The expression is: Among them, w1 represents the width of the feature map with lower resolution, and h1 represents the height of the feature map with lower resolution. Indicates whether the center of the g-th grid belongs to the parking space area. Indicates that the center of the g-th grid belongs to the parking space area, Indicates that the center of the g-th grid does not belong to the parking space area; x t represents the probability that the predicted parking space belongs to the tth type of parking space; y t Indicates the true value of the predicted parking space belonging to the tth parking space category, y t =(0,0,1) indicates a vertical parking space, y t =(0, 1, 0) indicates parallel parking space, y t =(1,0,0) indicates a tilted parking space, σ(x t ) is an intermediate function; Parking space occupancy loss function L O The expression is: Among them, α is the weight value for balancing effective parking spaces and invalid parking spaces. Indicates whether the g-th grid belongs to the valid parking space area. Indicates that the center point of the g-th grid is within the valid parking space area. Indicates that the center point of the g-th grid is located in the non-valid parking space area; Indicates whether the g-th grid belongs to the invalid parking space area. Indicates that the center point of the g-th grid is located in the invalid parking space area. Indicates that the center point of the g-th grid is located in the non-invalid parking space area; represents the probability of predicting whether the parking space on the g-th grid is valid, Indicates the true value of whether the predicted parking space on the g-th grid is valid, Indicates a valid parking space. Indicates an invalid parking space; Parking location confidence loss function L C The expression is: in, Indicates whether the g-th grid belongs to the parking space area, Indicates that the center of the g-th grid belongs to the parking space area, Indicates that the center of the g-th grid does not belong to the parking space area; represents the predicted probability value that the g-th grid belongs to a parking space; Indicates the true value that the g-th grid belongs to the parking space; α P The weight value for balancing parking areas and non-parking areas.

7. The method for detecting parking spaces for unmanned vehicles based on panoramic surround images according to claim 5, It is characterized in that The loss function L of the parking sign MP The expression is: in, Loss function representing the coordinates of parking space identification points The corresponding weight hyperparameter, W θ The loss function L representing the angle orientation of the parking space identification point θ The corresponding weight hyperparameters; Loss function of parking space identification point coordinates The expression is: Among them, w2 represents the width of the feature map with higher resolution, and h2 represents the height of the feature map with higher resolution. Indicates whether the g-th grid contains a parking space identification point. Indicates that the g-th grid contains the parking space identification point, Indicates that the g-th grid does not contain the parking space identification point; represents the predicted value of the x-coordinate direction of the parking space identification point on the g-th grid, Represents the true value of the x-coordinate direction of the parking space identification point on the g-th grid; represents the predicted value of the y-coordinate direction of the parking space identification point on the g-th grid, Represents the true value of the y-coordinate direction of the parking space identification point on the g-th grid; The loss function L of the parking space marking point angle orientation θ The expression is: in, Represents the cosine value of the angle orientation of the predicted parking space identification point on the g-th grid, Indicates the true value of the cosine value of the parking space identification point angle on the g-th grid; Represents the sin value of the angle orientation of the predicted parking space identification point on the g-th grid, Represents the true value of the sin value of the angle orientation of the parking space identification point on the g-th grid.

Citation Information

Patent Citations

  • Automatic parking space recognition panoramic look-around image real-time splicing method based on look-around

    CN111369439A

  • Parking space detection method based on panoramic video

    CN113468991A

  • Automatic parking sensing method integrating look-around image and laser radar

    CN114283391A

  • Vehicle four-way fisheye camera panoramic splicing method

    CN115936995A

  • Unmanned parking space detection method based on panoramic look-around image

    CN116229426A

Cited By

  • AI model construction system and method for unmanned aerial vehicle image recognition

    CN119888545A

  • Unmanned aerial vehicle image splicing method based on LightGlue and adaptive multiband fusion

    CN122175780A