A real-time splicing method for split flying vehicle bird's-eye view and parking space detection

Abstract

The present invention discloses a split-type flying vehicle bird's-eye view real-time splicing and parking space detection method. The method comprises the following steps: calibrating the internal and external parameters of a surround-view fisheye camera image data to obtain the internal parameters, distortion vector and homography transformation matrix of the fisheye camera; performing surround-view camera bird's-eye view splicing processing on the dedistorted image data to obtain a bird's-eye view centered on the flying vehicle; based on the surround-view camera bird's-eye view splicing result, using a deep learning-based method, feeding the spliced ​​bird's-eye view into a convolutional neural network to obtain the coordinates of two corner points of a detected parking space entrance line, and then inferring the coordinates of the other two corner points of the parking space using constraint conditions and prior information. The obtained parking space corner point coordinates are used to intercept the parking space in the bird's-eye view, and HOG feature extraction is performed on the parking space image to determine whether the parking space is vacant. Different colors are used to represent vacant parking spaces and occupied parking spaces, thereby realizing the flying vehicle bird's-eye view splicing and parking space detection.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned vehicles, and in particular relates to a method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection. Background Art

[0002] A fisheye camera is a camera with a fisheye lens, which has an extremely short focal length and a viewing angle close to or equal to 180°. Lenses with focal lengths of 16mm or shorter are considered extremely wide-angle lenses. Precisely because of this wide viewing angle, only four fisheye cameras are needed to stitch a bird's-eye view. Camera calibration uses specific methods to determine the unknown quantities in the imaging model (five intrinsic parameters, six extrinsic parameters, and distortion parameters). There are two main types of camera calibration: traditional calibration methods and self-calibration methods. Traditional calibration methods require a calibration reference object with known parameters. These methods, such as the direct linear transform (DLT) method, the Tsai two-step calibration method, and the Zhang Zhengyou planar calibration method, determine the camera parameters. While these traditional methods are relatively complex, they offer high accuracy. A bird's-eye view (BEV) is a bird's-eye view, characterized by a wide field of view and clear contrast. From the perspective of autonomous driving, we are more concerned with objects (especially cars and people) on the same plane as the vehicle, as well as their relationship to the road. A vertical view from above allows for uniformly proportioned object outlines in both the x and y dimensions, making them easier to display and analyze. Convolutional Neural Networks (CNNs) are a type of feedforward neural network with a deep structure that incorporates convolutional computations and is a representative algorithm for deep learning. CNNs possess representation learning capabilities, enabling shift-invariant classification of input information based on their hierarchical structure. When CNNs are used for object detection, an image is fed into the neural network, and the output is information such as the location, width, category, and confidence score of the predicted box. In this case, parking space detection outputs the location coordinates, shape, orientation, and confidence score of parking space corner points. The Histogram of Oriented Gradient (HOG) feature is a feature descriptor used for object detection in computer vision and image processing. HOG features are constructed by calculating and counting the gradient direction histogram of the local area of ​​the image. Its essence is the statistical information of the gradient, and the gradient mainly exists at the edge.

[0003] Patent CN201910549926.9 provides a method for generating a panoramic bird's-eye view from a vehicle-mounted surround-view camera. A calibration scene is established based on the vehicle; multiple vehicle-mounted surround-view cameras are provided on the vehicle; a corresponding checkerboard is set in the calibration scene based on the cameras; multiple corner points are provided in the checkerboard; a homography transformation matrix from each camera image to a bird's-eye view is obtained based on the calibration scene and the camera's internal parameters; the multiple camera images are converted into a bird's-eye view using the homography transformation matrix; the overlapping areas in the bird's-eye view are fused, and a spliced ​​surround-view bird's-eye view is output.

[0004] The homography transformation matrix of the existing technology has many constraints. For example, a calibration scene needs to be set in advance, one side of the checkerboard needs to be aligned with the placeholder frame, a large number of checkerboards are required, each camera needs to correspond to two checkerboards, and the spacing between the two checkerboards corresponding to each camera is equal, and the checkerboards need to be symmetrically distributed.

[0005] The first existing technology uses a fusion stitching method when stitching bird's-eye view images. Although this method can achieve a good fusion effect, it will increase the amount of calculation, prolong the stitching time, and affect the real-time performance of the bird's-eye view stitching.

[0006] Patent CN202011230752.9 provides a parking space detection method, which generates a bird's-eye view image based on images of at least two directions outside the vehicle; performs parking space detection on the bird's-eye view image with the parking space as the overall target, and determines the positioning information of the detected parking space.

[0007] The second existing technology performs parking space detection with the parking space as the overall target. Due to the limited field of view of the bird's-eye view, if the entire parking space is required for detection, the distance between the vehicle and the parking space is relatively close, which is not conducive to automatic parking and has a greater constraint on the detection distance.

[0008] The second existing technology requires multiple deep neural network models when detecting parking spaces of different shapes. Each model corresponds to a parking space type. When performing automatic parking, the corresponding model needs to be selected for parking space detection, and the model complexity is high. Summary of the Invention

[0009] In order to solve the technical problems existing in the background technology, the present invention aims to provide a method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection.

[0010] In order to solve the technical problem, the technical solution of the present invention is:

[0011] A method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection, the method comprising:

[0012] Calibrate the camera's intrinsic and extrinsic parameters for the surround-view fisheye camera image data to obtain the fisheye camera's intrinsic parameters, distortion vector, and homography transformation matrix.

[0013] Based on the intrinsic parameters and distortion vector of the fisheye camera, the surround-view fisheye camera image data is dedistorted to obtain dedistorted image data;

[0014] The dedistorted image data is stitched with the surround camera bird's-eye view based on the homography transformation matrix to obtain a bird's-eye view centered on the flying vehicle.

[0015] The bird's-eye view centered on the flying vehicle is fed into the convolutional neural network of the preset parking space detection model to obtain the coordinates of the two corner points of the parking space entrance line; the preset parking space detection model retains the backbone of MobileNetv3, removes the subsequent pooling layer and fully connected layer, and uses a convolutional layer to adjust the network output to the desired scale and dimension.

[0016] Based on the coordinates of the two corner points of the parking space entrance line, the coordinates of the two corner points at the end of the parking space are inferred using the preset constraints and prior information;

[0017] The obtained coordinates of the two corner points of the parking space entrance line and the two corner points of the parking space end are used to intercept the parking space in the bird's-eye view centered on the flying vehicle. HOG feature extraction is performed on the intercepted parking space image to determine whether the parking space is vacant. Different colors are used to represent vacant and occupied parking spaces, thus realizing the bird's-eye view splicing and parking space detection of the flying vehicle.

[0018] Furthermore, the surround fisheye camera image data is collected by the fisheye camera.

[0019] Furthermore, the image data of the surround-view fisheye camera is obtained, and the camera intrinsic parameter calibration is performed on the surround-view fisheye camera image data to obtain the fisheye camera intrinsic parameter K and distortion vector D. The image dedistortion processing can be completed, which specifically includes:

[0020] Find the checkerboard corner points in the surround-view fisheye camera image data and obtain the coordinates of the checkerboard corner points;

[0021] Performing sub-pixel optimization on the checkerboard corner point coordinates to obtain optimized checkerboard corner point coordinates;

[0022] Based on the optimized checkerboard corner coordinates, the Zhang Zhengyou calibration method is used to estimate and calculate the camera intrinsic parameters to obtain the calibration results, and the calibration results are continuously optimized based on the new image data;

[0023] θ d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 );

[0024] Among them, k1, k2, k3, k4 are the fisheye camera distortion parameters D. The distortion of the fisheye camera is described by the θ polynomial with k1, k2, k3, k4 as coefficients;

[0025]

[0026] Among them, f x and f y is the focal length of the fisheye camera, c x and c y It is the relative displacement of the origin of the pixel coordinate system relative to the origin of the physical coordinate system. The camera intrinsic parameter matrix is ​​represented by K.

[0027] The mapping matrix is ​​obtained using the camera intrinsic parameter K and the distortion parameter D, that is, the distortion-free and corrected transformation relationship is obtained;

[0028] Based on the undistorted and corrected transformation relationship, the surround-view fisheye camera image data is remapped to achieve dedistortion processing.

[0029] Furthermore, the camera is calibrated with external parameters, including:

[0030] Using a fisheye camera installed on each side of the flying vehicle and a calibration plate placed in front of each camera, the original image without distortion is obtained;

[0031] Performing image distortion correction on the original image according to the camera intrinsic parameter K and the distortion vector D to obtain a dedistorted image;

[0032] Use a drone or photography equipment from a higher vantage point to obtain a bird's-eye view of the vehicle and the four calibration plates.

[0033] preprocessing the bird's-eye view to obtain a preprocessed bird's-eye view;

[0034] The dedistorted image is the original image, and the preprocessed bird's-eye view is the target image. Calibration plate corner point detection is performed on the original image and the target image to obtain the coordinates of the corner points of the calibration plate under two viewing angles;

[0035] The corner coordinates of the original image and the target image are used to obtain the homography transformation matrix H from the distorted image to the preprocessed bird's-eye view image.

[0036] Furthermore, the preprocessing specifically includes: rotating the bird's-eye view so that the front of the vehicle is located directly in front and remains vertical, cropping the bird's-eye view ratio and adjusting the bird's-eye view resolution, and centering and scaling the adjusted bird's-eye view so that the vehicle is in the center of the bird's-eye view.

[0037] Furthermore, the bird's-eye view image of the surround camera is stitched, which specifically includes:

[0038] Using the homography transformation matrix H, a bird's-eye view of the dedistorted image is obtained;

[0039] For the bird's-eye view converted from the dedistorted image, choose direct stitching or fusion stitching to obtain a bird's-eye view centered on the flying vehicle;

[0040] If fusion stitching is selected, RGB values ​​are fused based on the distance from the pixel to the boundary of the overlapping area;

[0041]

[0042] P=σP A +(1-σ)P B ;

[0043] in, is the angle between the pixel and the boundary of the overlapping area, is the angle of the overlapping area, P A and P B is the RGB value of the pixel in the overlapping area, and P is the RGB value of the point after fusion;

[0044] The bird's-eye view centered on the flying vehicle is fed into a convolutional neural network in a preset parking space detection model to obtain the coordinates and orientation information of the two corner points of the parking space entrance line.

[0045] Furthermore, the coordinates of the two corner points at the end of the parking space are inferred using the preset constraints and prior information; specifically,

[0046] Filter out pairs of parking space entrance line corner points based on distance constraints;

[0047] t1≤‖P1P2‖≤t2;

[0048] Among them, t1 and t2 are the range of the parking space entrance width, which is a priori information, and P1 and P2 are the two corner points of the parking space entrance line;

[0049] For corner points that meet the distance constraint, it is also necessary to determine whether the pair of corner points can form a valid parking space entrance line. Then, the parking space is classified based on the paired corner points. The type of parking space is determined based on the direction information of the two corner points. Whether the parking space is a vertical parking space, a parallel parking space, an acute angle parking space, or an obtuse angle parking space, as well as the angle of the parking space, is determined.

[0050] Using the parking space angle and the coordinates of the two corner points of the entrance line, the coordinates of the two corner points at the end of the parking space are obtained;

[0051]

[0052]

[0053] Among them, P1, P2, P3 and P4 are the four corner points of the parking space, d is the depth of the parking space, which is prior information, and α is the angle of the parking space.

[0054] Furthermore, HOG feature extraction is performed on the captured parking space image to determine whether the parking space is vacant, specifically including:

[0055] Performing HOG feature extraction on the parking space image to obtain an image after feature extraction;

[0056] Performing variance calculation on the image after feature extraction to determine whether a parking space is vacant;

[0057] var<0.083;

[0058] Here, var is the variance value of the image, and 0.083 is the threshold for determining whether a parking space is vacant, which is prior information. If the variance is less than 0.083, the parking space is vacant; otherwise, there is a vehicle.

[0059] A split-type flying vehicle bird's-eye view real-time splicing and parking space detection system, the system comprising:

[0060] one or more processors;

[0061] a memory for storing one or more programs;

[0062] When the one or more programs are executed by the one or more processors, the one or more processors execute any of the above-mentioned methods for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection.

[0063] Compared with the prior art, the advantages of the present invention are:

[0064] When calibrating camera extrinsic parameters, there are no restrictions on the placement of the calibration plate. It is sufficient that the captured image can correctly identify the corner points of the calibration plate in the image. Only four calibration plates are needed to be placed in four directions to complete the calibration, reducing the complexity of the calibration operation and reducing costs.

[0065] In the process of stitching the bird's-eye view, you can choose to use direct stitching or fusion stitching, and flexibly select the stitching method according to the application scenario. If higher real-time performance is required, use direct stitching. If higher stitching quality is required, use fusion stitching.

[0066] Using a deep neural network to detect parking space corners, the four-point coordinates of the parking space are inferred based on prior parking space information, significantly increasing the detection distance and eliminating the need for an image of the entire parking space. Furthermore, the corner detection method uses only a single network model, which can be used through post-processing to infer the four-point coordinates of different parking space types, reducing model complexity.

[0067] By using the HOG feature extraction method, it is possible to determine whether a parking space is vacant while performing parking space detection, thus ensuring real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 , a flow chart of a method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection provided by an embodiment of the present invention;

[0069] Figure 2 , an example of a bird's-eye view required for extrinsic calibration of a fisheye camera provided by an embodiment of the present invention;

[0070] Figure 3 , a schematic diagram of a checkerboard used for calibration provided in an embodiment of the present invention;

[0071] Figure 4 , a flow chart of fisheye camera calibration provided by an embodiment of the present invention;

[0072] Figure 5 , a schematic diagram of the fusion splicing principle provided by an embodiment of the present invention;

[0073] Figure 6 , an example of a spliced ​​bird's-eye view provided by an embodiment of the present invention;

[0074] Figure 7 , a schematic diagram of the parking space coordinate reasoning principle provided by an embodiment of the present invention;

[0075] Figure 8 , a flowchart of HOG feature extraction of parking space images provided by an embodiment of the present invention;

[0076] Figure 9 , a schematic diagram of parking space detection from a bird's-eye view provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The specific implementation of the present invention is described below in conjunction with examples:

[0078] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the efficacy and purpose that can be achieved by the present invention.

[0079] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0080] Example

[0081] The method for stitching a bird's-eye view using a surround-view fisheye camera and detecting a parking space provided in an embodiment of the present invention is applied to a split flying vehicle. Multiple fisheye cameras are arranged on the split flying vehicle to capture images in real time for stitching a bird's-eye view, and parking space detection is performed on the stitched bird's-eye view to obtain position information of the parking space relative to the flying vehicle, thereby providing position information for automatic parking of the flying vehicle, which is beneficial for docking with the aircraft after parking.

[0082] Figure 1 A flow chart of a method for real-time splicing of a split flying vehicle bird's-eye view and parking space detection provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the method includes:

[0083] Step 101: Calibrate the internal and external parameters of the fisheye camera.

[0084] Multiple surround-view cameras are placed around the split-body flying vehicle. These can be fisheye cameras or standard cameras. In this example, fisheye cameras are used, so only four are needed to achieve 360-degree surround view. Before performing parameter calibration, the resolution and frame rate of the selected cameras must be determined.

[0085] Figure 3 A checkerboard diagram is provided for an embodiment of the present invention. In this example, the checkerboard calibration plate used is 600mm*600mm in size, with 8*8 corner points and a single black and white square of 60mm*60mm in size. The first step is internal parameter calibration. The camera internal parameters refer to the camera's preset internal parameter matrix and distortion parameters. First, use a fisheye camera to take multiple pictures of the checkerboard calibration plate. The calibration plate needs to be in different positions in the pictures, and the corner points of the calibration plate need to be clearly visible. Then, use the Zhang Zhengyou calibration method to perform internal parameter calibration. A sufficient number of valid pictures are required to obtain a good calibration effect. Multiple calibrations can be performed to improve calibration accuracy. After calibration is completed, the camera internal parameter K and distortion coefficient D can be obtained.

[0086] Figure 4 The fisheye camera calibration flow chart provided in the embodiment of the present invention is as follows: after completing the internal parameter calibration, the external parameter calibration can be performed. Figure 2As shown, place a calibration plate separately, ensuring the lighting and distance between the plates are correct. This ensures the plates are clear and easy to detect corners in the surround-view camera. Obtain camera images from four viewpoints and generate dedistorted images using the camera intrinsic parameters K and distortion coefficient D. Then, use a drone or a camera from a high vantage point to capture a bird's-eye view, ensuring that the four calibration plates are aligned horizontally in the bird's-eye view. The captured bird's-eye view requires manual processing: rotate the image to position the vehicle's front directly in front and maintain a vertical orientation. Crop the image and adjust the image resolution. This resolution will be used for the final stitched bird's-eye view image. Finally, center and resize the adjusted image to center the vehicle. These operations ensure the correct orientation of the final stitched image. When calculating the homography matrix using the dedistorted image and the bird's-eye view, mask the three calibration plates in the bird's-eye view. For example, when calculating the homography matrix for the front-view camera, black out the left, right, and rear calibration plates in the bird's-eye view. The corner points of the calibration plate are detected for each of the four groups of images (dedistorted images and masked bird's-eye view images), and the homography transformation matrix H is calculated.

[0087] Step 102: Real-time stitching of the bird's-eye view.

[0088] The bird's-eye view is created by reprojecting the pixels of the camera image plane onto the horizontal ground. Specifically, a homography is used to transform the dedistorted image into a bird's-eye view. After completing the transformation of the dedistorted images in four directions, the four bird's-eye view images are aligned with the camera's orientation. For example, the bird's-eye view of the front-facing camera only has an image in the upper half; the rest is black.

[0089] After obtaining the bird's-eye view from four directions, you can stitch them together. You can choose to use direct stitching or fusion stitching. You can flexibly choose the stitching method according to the application scenario. If you need higher real-time performance, use direct stitching. If you need higher stitching quality, use fusion stitching. When stitching directly, capture the four corners of the vehicle image and the bird's-eye view. Figure 4 The images between the lines of the corners are used to obtain the corresponding bird's-eye view. The vehicle images here can be replaced according to your needs. Just select the appropriate resolution and place the intercepted bird's-eye view images from all directions in the corresponding positions to complete the stitching. Figure 6 The following is an example of a bird's-eye view image after direct stitching. When fusion stitching is performed, the weight of the RGB value of the overlapping area of ​​adjacent bird's-eye view images needs to be calculated based on the distance. Figure 5 As shown in the figure, the RGB values ​​of each pixel in the overlapping area are fused according to the following algorithm, and finally a fused bird's-eye view is obtained.

[0090]

[0091] P=σP A +(1-σ)P B ;

[0092] in, is the angle between the pixel and the boundary of the overlapping area, is the angle of the overlapping area, P A and P B is the RGB value of the pixel in the overlapping area, and P is the RGB value of the pixel after fusion. According to the algorithm, the closer the distance to A is, the larger the proportion of the RGB value of the pixel, and vice versa.

[0093] If real-time bird's-eye view stitching is required, the fisheye camera image can be captured at a certain time interval and the original image can be replaced. For example, a picture can be captured every 15 frames. Then, the camera intrinsic parameter K, distortion coefficient D and homography transformation matrix H can be used to repeat the above steps to achieve real-time bird's-eye view stitching.

[0094] Step 103: parking space detection and parking space reasoning.

[0095] The parking space detection method in this example is based on a deep neural network method. It is necessary to train the neural network model in advance and save the model weights with the best detection effect. The spliced ​​bird's-eye view image is input into the network model to obtain the position coordinates, direction, shape and confidence of the corner points of the parking space entrance line in the bird's-eye view image. The shape here refers to the corner points being divided into L-shaped and T-shaped. The direction refers to the direction defined by the corner point. The pattern of the T-shaped corner point is symmetrical, and we define its direction as the direction along the symmetry axis. For the L-shaped corner point, the defined direction is one side of the L, which coincides with the other side after being rotated 90 degrees. After obtaining the corner points in the bird's-eye view, we still need to filter them. Using the distance constraint, the distance between the corner points of the parking space entrance line needs to satisfy the following formula.

[0096] t1≤‖P1P2‖≤t2;

[0097] Among them, t1 and t2 are the value ranges of the parking space entrance width, which are prior information, and P1 and P2 are the two corner points of the parking space entrance line.

[0098] The corner points that meet the distance constraint also need to determine whether the pair of corner points can form a valid parking space. Get the paired entrance line corner points that can form a parking space in the bird's-eye view, and use the paired corner points to obtain the direction and depth information of the parking space. Figure 7 The following formula is used to infer the other two corner points of the parking space based on the principle diagram of the parking space coordinates provided in the embodiment of the present invention.

[0099]

[0100]

[0101] Among them, P1, P2, P3 and P4 are the four corner points of the parking space, d is the depth of the parking space, which is prior information, and α is the angle of the parking space.

[0102] Step 104: Determine whether a parking space is available.

[0103] Figure 8 This is a flowchart for extracting HOG features from parking space images provided by an embodiment of the present invention. Based on the coordinates of the parking space, its position in the bird's-eye view is determined. If the coordinate point exceeds the image frame, the coordinate point is corrected to the intersection of the parking space line and the image boundary. After obtaining the corrected coordinates of the four corner points of the parking space in the bird's-eye view, the portion outside the parking space is set to black, that is, the RGB value is (255, 255, 255). The coordinates of the parking space's circumscribed rectangle are obtained, and the parking space is converted into the shape of the circumscribed rectangle through perspective transformation. The image is cropped to retain the parking space portion and to the same image frame size to ensure that the size of each parking space is consistent and reduce the impact on subsequent parking space vacancy determination. After obtaining a uniformly sized parking space image, HOG feature extraction is performed to generate a parking space HOG image. If a vehicle is parked in the parking space, the distribution of the HOG features is relatively chaotic, while if there is no vehicle, the distribution is uniform. Therefore, by calculating the variance, when the variance is less than a certain threshold, the parking space is vacant; otherwise, a vehicle is parked. The variance threshold in this example is 0.083, which was determined through experiments. In this way, we can set the parking space color to show the parking spaces with and without cars, such as Figure 9 shown.

[0104] References (such as patents / papers / standards)

[0105] [1] Zhang Rong, Xiong Qi, Zhang Fang, et al. A method for generating panoramic bird's-eye view images from a vehicle-mounted surround view camera: CN110288527A[P]. 2019.

[0106] [2] Ji Fangcheng, Tan Richeng, Hou Wenbo. A parking space detection method and device, automatic parking method and device:, CN112298168A[P]. 2021.

[0107] [3]Huang J, Zhang L, Shen Y, et al.DMPR-PS: A Novel Approach for Parking-Slot Detection Using Directional Marking-Point Regression[C] / / 2019IEEEInternational Conference on Multimedia and Expo(ICME).IEEE, 2019.

[0108] [4]Li W,Cao L,Yan L,et al.Vacant Parking Slot Detection in the AroundView Image Based on Deep Learning[J].Sensors(Basel,Switzerland),2020,20(7).

[0109] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

[0110] Many other changes and modifications can be made without departing from the spirit and scope of the present invention. It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection, characterized in that: The method comprises: Calibrate the camera's intrinsic and extrinsic parameters for the surround-view fisheye camera image data to obtain the fisheye camera's intrinsic parameters, distortion vector, and homography transformation matrix. Based on the intrinsic parameters and distortion vector of the fisheye camera, the surround-view fisheye camera image data is dedistorted to obtain dedistorted image data; The dedistorted image data is stitched with the surround camera bird's-eye view based on the homography transformation matrix to obtain a bird's-eye view centered on the flying vehicle. The aerial view centered on the flying vehicle is fed into a convolutional neural network in a preset parking space detection model to obtain the coordinates of two corner points of the parking space entrance line; Based on the coordinates of the two corner points of the parking space entrance line, the coordinates of the two corner points at the end of the parking space are inferred using the preset constraints and prior information; The obtained coordinates of the two corner points of the parking space entrance line and the two corner points of the parking space end are used to intercept the parking space in the bird's-eye view centered on the flying vehicle. HOG feature extraction is performed on the intercepted parking space image to determine whether the parking space is vacant. Different colors are used to represent vacant and occupied parking spaces, thus realizing the bird's-eye view splicing and parking space detection of the flying vehicle.

2. A method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection according to claim 1, characterized in that: The fisheye camera image data is collected through the fisheye camera.

3. The method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection according to claim 1, characterized in that: Obtain the surround-view fisheye camera image data, perform camera intrinsic calibration on the surround-view fisheye camera image data, obtain the fisheye camera intrinsic parameter K and distortion vector D, and then complete the image dedistortion processing, which specifically includes: Find the checkerboard corner points in the surround-view fisheye camera image data and obtain the coordinates of the checkerboard corner points; Performing sub-pixel optimization on the checkerboard corner point coordinates to obtain optimized checkerboard corner point coordinates; Based on the optimized checkerboard corner coordinates, the Zhang Zhengyou calibration method is used to estimate and calculate the camera intrinsic parameters to obtain the calibration results, and the calibration results are continuously optimized based on the new image data; i d =θ(1+k1θ 2 +k2θ 4 +k3θ 6 +k4θ 8 ); Among them, k1, k2, k3, k4 are the distortion parameters D of the fisheye camera. The distortion of the fisheye camera is described by the θ polynomial with k1, k2, k3, k4 as coefficients; Among them, f x and f y is the focal length of the fisheye camera, c x and c y It is the relative displacement of the origin of the pixel coordinate system relative to the origin of the physical coordinate system. The camera intrinsic parameter matrix is ​​represented by K. The mapping matrix is ​​obtained using the camera intrinsic parameter K and the distortion parameter D, that is, the distortion-free and corrected transformation relationship is obtained; Based on the undistorted and corrected transformation relationship, the surround-view fisheye camera image data is remapped to achieve dedistortion processing.

4. The method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection according to claim 3 is characterized in that: Calibrate the camera's external parameters, including: Using a fisheye camera installed on each side of the flying vehicle and a calibration plate placed in front of each camera, the original image without distortion is obtained; Performing image distortion correction on the original image according to the camera intrinsic parameter K and the distortion vector D to obtain a dedistorted image; Use a drone or photography equipment from a higher vantage point to obtain a bird's-eye view of the vehicle and the four calibration plates. preprocessing the bird's-eye view to obtain a preprocessed bird's-eye view; The dedistorted image is the original image, and the preprocessed bird's-eye view is the target image. Calibration plate corner point detection is performed on the original image and the target image to obtain the coordinates of the corner points of the calibration plate under two viewing angles; The corner coordinates of the original image and the target image are used to obtain the homography transformation matrix H from the distorted image to the preprocessed bird's-eye view image.

5. The method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection according to claim 4, characterized in that: The preprocessing specifically includes: rotating the bird's-eye view so that the front of the vehicle is located directly in front and remains vertical, cropping the bird's-eye view ratio and adjusting the resolution of the bird's-eye view, and centering and scaling the adjusted bird's-eye view so that the vehicle is located in the center of the bird's-eye view.

6. The method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection according to claim 4, characterized in that: Perform surround camera bird's-eye view stitching, including: Using the homography transformation matrix H, a bird's-eye view of the dedistorted image is obtained; For the bird's-eye view converted from the dedistorted image, choose direct stitching or fusion stitching to obtain a bird's-eye view centered on the flying vehicle; If fusion stitching is selected, RGB values ​​are fused based on the distance from the pixel to the boundary of the overlapping area; P=σP A +(1-σ)P B ; in, is the angle between the pixel and the boundary of the overlapping area, is the angle of the overlapping area, P A and P B is the RGB value of the pixel in the overlapping area, and P is the RGB value of the point after fusion; The bird's-eye view centered on the flying vehicle is fed into a convolutional neural network in a preset parking space detection model to obtain the coordinates and orientation information of the two corner points of the parking space entrance line.

7. The method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection according to claim 6, characterized in that: The coordinates of the two corner points at the end of the parking space are inferred using the preset constraints and prior information; specifically: Filter out pairs of parking space entrance line corner points based on distance constraints; t1≤||P1P2||≤t2; Among them, t1 and t2 are the range of the parking space entrance width, which is a priori information, and P1 and P2 are the two corner points of the parking space entrance line; For corner points that meet the distance constraint, it is also necessary to determine whether the pair of corner points can form a valid parking space entrance line. Then, the parking space is classified based on the paired corner points. The type of parking space is determined based on the direction information of the two corner points. Whether the parking space is a vertical parking space, a parallel parking space, an acute angle parking space, or an obtuse angle parking space, as well as the angle of the parking space, is determined. Using the parking space angle and the coordinates of the two corner points of the entrance line, the coordinates of the two corner points at the end of the parking space are obtained; Among them, P1, P2, P3 and P4 are the four corner points of the parking space, d is the depth of the parking space, which is prior information, and α is the angle of the parking space.

8. The method for real-time splicing of a split-type flying vehicle bird's-eye view and parking space detection according to claim 7, characterized in that: Perform HOG feature extraction on the captured parking space image and determine whether the parking space is vacant, specifically including: Performing HOG feature extraction on the parking space image to obtain an image after feature extraction; Performing variance calculation on the image after feature extraction to determine whether a parking space is vacant; var<0.083; Here, var is the variance value of the image, and 0.083 is the threshold for determining whether a parking space is vacant, which is prior information. If the variance is less than 0.083, the parking space is vacant; otherwise, there is a vehicle.

Images