Parking space identification method, electronic equipment and medium

Abstract

The invention provides a parking space identification method, electronic equipment and a medium, and the method comprises the steps: obtaining each key frame group in a video frame collected by a camera for each camera, and determining each first feature point corresponding to each key frame group and a second feature point corresponding to each first feature point for each key frame group; determining a camera pose of the camera according to the first feature points and the second feature points corresponding to the first feature points; sparse point clouds corresponding to the first feature points are determined according to the camera pose of the camera and the first feature points; according to each point cloud point in each sparse point cloud corresponding to each camera, determining three-dimensional Gaussian distribution, performing voxelization processing on the three-dimensional Gaussian distribution, and determining each voxel and an occupation value corresponding to each voxel; and determining a parking space detection frame corresponding to the key frame group according to the occupancy value of each voxel and a pre-trained three-dimensional point cloud parking space detection model, thereby realizing parking space identification in a three-dimensional space, and improving identification accuracy of a non-planar parking space.

Description

Technical Field

[0001] The present application relates to the field of assisted driving technology, and in particular to a parking space recognition method, electronic equipment and medium. Background Art

[0002] Parking space perception is one of the key functions of the automatic parking assistance system. Automatic parking usually includes key steps such as the vehicle finding a suitable parking space in the parking lot, performing precise parking actions, and avoiding collisions with surrounding obstacles. In this process, parking space perception plays a vital role. Traditional parking space perception algorithms usually use fisheye cameras to build AVM (Around View Monitor) and then perform parking space frame perception on the surround view image.

[0003] However, this method usually assumes that the scenes captured by the four cameras are usually on the same plane. When the vehicle drives on a steep slope, the surround view images formed by the four fisheye cameras will be misaligned, and there will be a large error between the identified parking space and the actual parking space, which will affect the process of automatic parking. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, the present application aims to provide a parking space recognition method, electronic device and medium to realize the recognition of parking spaces in three-dimensional space and improve the recognition accuracy of non-planar parking spaces.

[0005] The present invention provides a method for identifying a parking space, the method comprising:

[0006] For each camera, each key frame group in the video frame captured by the camera is obtained, and for each key frame group, each first feature point corresponding to the key frame group and a second feature point corresponding to each first feature point are determined; wherein the key frame group is a combination of every two adjacent key frames in the video frame;

[0007] Determining a camera pose of the camera according to each first feature point and a second feature point corresponding to each first feature point;

[0008] Determine, according to the camera pose of the camera and each first feature point, a sparse point cloud corresponding to each first feature point;

[0009] Determine a three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera, and voxelize the three-dimensional Gaussian distribution to determine each voxel and an occupancy value corresponding to each voxel;

[0010] According to the occupancy value of each voxel and the pre-trained three-dimensional point cloud parking space detection model, the parking space detection frame corresponding to the key frame group is determined.

[0011] According to the technical solution provided in the embodiment of the present application, optionally, determining the camera pose of the camera according to each first feature point and a second feature point corresponding to each first feature point includes:

[0012] Determine an initial relative pose based on the first feature points and the second feature points corresponding to the first feature points and the structure from motion algorithm;

[0013] Acquire motor speed and inertial measurement information, and determine Kalman relative posture according to the motor speed and the inertial measurement information;

[0014] The camera pose of the camera is determined according to the initial relative pose and the Kalman relative pose.

[0015] According to the technical solution provided in the embodiment of the present application, optionally, after determining the camera pose of the camera according to the initial relative pose and the Kalman relative pose, the method further includes:

[0016] According to the feature points in the key frames of each camera at consecutive preset moments, and the three-dimensional points corresponding to each feature point, the preset objective function is minimized and the camera posture of each camera is updated; wherein the preset objective function is constructed based on the error between the conversion points corresponding to each three-dimensional point observed by each camera and the feature points in the key frames of each camera at each preset moment.

[0017] According to the technical solution provided in the embodiment of the present application, optionally, the preset objective function is:

[0018]

[0019] Among them, P i is the i-th 3D point among the 3D points observed by all cameras, N is the total number of 3D points, M is the total number at the preset time, C is the total number of cameras, is the effective value of the i-th 3D point observed by the c-th camera at the t-th time, m itc is the feature point corresponding to the i-th 3D point of the c-th camera at the t-th time, ||·|| huber is the smooth L1 norm, is the preset external parameter transformation matrix of the c-th camera, is the homogeneous transformation matrix to be solved, π c is the local camera model of the c-th camera, is the transformation point corresponding to the i-th 3D point, F 1c F 2c is the camera pose of the c-th camera.

[0020] According to the technical solution provided in the embodiment of the present application, optionally, determining the three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera includes:

[0021] According to the initial parameters, each point cloud point in each sparse point cloud corresponding to each camera is converted into a single-point Gaussian distribution, and the initial Gaussian distribution is determined according to each single-point Gaussian distribution;

[0022] Determine a rasterized image corresponding to each camera according to the initial Gaussian distribution and the camera pose of each camera;

[0023] According to the rasterized images corresponding to the cameras and the key frames corresponding to the rasterized images, each single-point Gaussian distribution is updated based on a preset loss function, and a three-dimensional Gaussian distribution is determined according to the updated single-point Gaussian distributions.

[0024] According to the technical solution provided in the embodiment of the present application, optionally, the three-dimensional Gaussian distribution is voxelized to determine each voxel and the occupancy value corresponding to each voxel, including:

[0025] According to a preset resolution, the three-dimensional Gaussian distribution is voxelized to obtain each voxel;

[0026] For each voxel, the occupancy value of the voxel is determined according to each updated single-point Gaussian distribution corresponding to the voxel.

[0027] According to the technical solution provided in the embodiment of the present application, optionally, determining the occupancy value of the voxel according to each updated single-point Gaussian distribution corresponding to the voxel includes:

[0028] The occupancy value of the voxel is determined based on the following formula:

[0029]

[0030] Among them, v j is the occupancy value of the jth voxel, N is the number of updated single-point Gaussian distributions corresponding to the voxel, α i is the opacity of the updated i-th single-point Gaussian distribution corresponding to the voxel, β i is the probability density of the updated i-th single-point Gaussian distribution corresponding to the voxel.

[0031] According to the technical solution provided in the embodiment of the present application, optionally, determining the parking space detection frame corresponding to the key frame group according to the occupancy value of each voxel and the pre-trained three-dimensional point cloud parking space detection model includes:

[0032] For each voxel, in response to the occupancy value of the voxel being greater than a preset value, the occupancy value of the voxel is updated to 1; in response to the occupancy value of the voxel being less than or equal to the preset value, the occupancy value of the voxel is updated to 0;

[0033] The updated occupancy value of each voxel is input into a pre-trained three-dimensional point cloud parking space detection model to obtain a parking space detection frame corresponding to the key frame group.

[0034] The present application also provides an electronic device, the electronic device comprising:

[0035] Processor and memory;

[0036] The processor is used to execute the steps of the parking space recognition method as described in any embodiment by calling the program or instruction stored in the memory.

[0037] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute the steps of the parking space recognition method as described in any embodiment.

[0038] In summary, the present application proposes a parking space recognition method, by obtaining each key frame group in the video frame captured by the camera for each camera, determining each first feature point corresponding to the key frame group and the second feature point corresponding to each first feature point for each key frame group, and then determining the camera pose of the camera according to each first feature point and the second feature point corresponding to each first feature point, so as to facilitate subsequent conversion in three-dimensional space, further, according to the camera pose of the camera and each first feature point, determining the sparse point cloud corresponding to each first feature point, and determining the three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera, so as to obtain a parking space recognition method. The planar image is converted into three-dimensional space, and further, the three-dimensional Gaussian distribution is voxelized to determine each voxel and the occupancy value corresponding to each voxel. According to the occupancy value of each voxel and the pre-trained three-dimensional point cloud parking space detection model, the parking space detection frame corresponding to the key frame group is determined, thereby realizing the effect of identifying parking spaces in three-dimensional space and improving the recognition accuracy of non-planar parking spaces. Compared with the prior art, the parking space recognition method in which each camera is on the same plane by default takes non-planar scenes into consideration, and can effectively identify parking spaces in three-dimensional space. It can also accurately identify parking spaces in scenes with large slopes, thereby improving the subsequent parking experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of a parking space identification method provided in an embodiment of the present application;

[0040] Figure 2is a flow chart of another parking space identification method provided in an embodiment of the present application;

[0041] Figure 3 is a processing flow chart of a parking space identification method provided in an embodiment of the present application;

[0042] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] Figure 1 is a flow chart of a parking space identification method provided by an embodiment of the present application. Figure 1 , the parking space recognition method specifically includes:

[0046] S110: For each camera, obtain each key frame group in the video frame captured by the camera, and for each key frame group, determine each first feature point corresponding to the key frame group and each second feature point corresponding to the first feature point.

[0047] Among them, in order to perform parking space recognition in three-dimensional space, this embodiment needs to use multiple cameras, such as four-way cameras, or more-way cameras, and the type of camera can be a fisheye camera. Video frames are continuous image frames captured by the camera. A key frame group is a combination of every two adjacent key frames in a video frame. A key frame is a video frame that meets the key frame requirements, and the key frame contains important states or key changes. The key frame requirement can be that the feature matching point pair (each first feature point and the second feature point corresponding to each first feature point) with the adjacent key frame is less than a threshold and the time is greater than a time threshold (such as 0.1s). The first feature point and the second feature point corresponding to the first feature point are feature points located in two different key frames in the key frame group, respectively, and have a matching relationship.

[0048] Specifically, for each camera, video frames are collected respectively, and key frames are extracted from the video frames. The combination of every two adjacent key frames is taken as a key frame group, and the feature point matching pairs corresponding to each key frame group can be determined. For example, the ORB (Oriented FAST and Rotated BRIEF) algorithm can be used to extract the first feature points and the second feature points corresponding to the first feature points.

[0049] S120: Determine the camera pose of the camera according to each first feature point and the second feature point corresponding to each first feature point.

[0050] Among them, the camera pose refers to the position and posture of the camera in the world coordinate system, which is usually determined by rotation and translation.

[0051] Specifically, according to the position calculation of each first feature point in the key frame group and the second feature point corresponding to each first feature point, the camera pose corresponding to the key frame group can be estimated. It can be understood that the first frame in each key frame can be used as the origin of the world coordinate system, and subsequent solutions can be performed on this basis.

[0052] Based on the above example, the camera pose can be estimated by combining the SFM (Structure from Motion) initialization method and the Extended Kalman Filter (EKF) algorithm. Specifically, the camera pose of the camera can be determined according to each first feature point and the second feature point corresponding to each first feature point in the following manner:

[0053] Determine an initial relative pose based on the first feature points and the second feature points corresponding to the first feature points and the structure from motion algorithm;

[0054] Obtain motor speed and inertial measurement information, and determine the Kalman relative posture based on the motor speed and inertial measurement information;

[0055] The camera pose is determined according to the initial relative pose and the Kalman relative pose.

[0056] Among them, the structure from motion algorithm is a 3D reconstruction method that estimates the geometric structure of the scene by obtaining corresponding points from multiple images. The initial relative pose is the result of the structure from motion algorithm to solve the camera pose. The motor speed is the number of revolutions per minute of the motor, which can be collected by the corresponding sensor. Inertial measurement information is based on information collected by the inertial measurement sensor (IMU), such as acceleration, angular velocity, and direction. Kalman relative pose is the result of the extended Kalman filter algorithm to solve the camera pose.

[0057] Specifically, the structure from motion algorithm is used to solve the camera pose for each first feature point and the second feature point corresponding to each first feature point to obtain the initial relative pose. The relative pose of two key frames in the key frame group can be obtained by using the EKF algorithm, which is the Kalman relative pose. The camera pose of the camera can be obtained by performing nonlinear optimization on the initial relative pose and the Kalman relative pose.

[0058] Based on the above example, after determining the camera pose according to the initial relative pose and the Kalman relative pose, it can also be optimized by multi-frame bundling adjustment, which can be:

[0059] According to the feature points in the key frames of each camera at consecutive preset moments and the 3D points corresponding to each feature point, the preset objective function is minimized and the camera pose of each camera is updated.

[0060] The preset moments are multiple consecutive moments, for example, 50 consecutive moments, etc. The specific number can be set according to the needs. The three-dimensional points are points in the three-dimensional space that can be captured by each camera, and have a corresponding relationship with the feature points in the key frame. The corresponding relationship can be obtained based on the motion recovery structure algorithm. The preset objective function is constructed based on the error between the conversion points corresponding to each three-dimensional point observed by each camera and each feature point in the key frame of each camera at each preset moment.

[0061] Specifically, each feature point in the key frame of each camera at each consecutive preset moment and the three-dimensional point corresponding to each feature point are input into a preset objective function to obtain the objective function value, and the camera pose of each camera is solved with the minimum objective function value as the goal. The camera pose of each camera is updated based on the camera pose obtained by the optimization solution.

[0062] Based on the above example, the preset objective function can be:

[0063]

[0064] Among them, P iis the i-th 3D point among the 3D points observed by all cameras, N is the total number of 3D points, M is the total number at the preset time, C is the total number of cameras, is the effective value of the i-th 3D point observed by the c-th camera at the t-th time, m itc is the feature point corresponding to the i-th 3D point of the c-th camera at the t-th time, ||·|| huber is the smooth L1 norm, is the preset external parameter transformation matrix of the c-th camera, is the homogeneous transformation matrix to be solved, π c is the local camera model of the c-th camera, is the transformation point corresponding to the i-th 3D point, F 1c F 2c is the camera pose of the c-th camera.

[0065] It can be understood that optimizing the camera pose using the above preset objective function is to optimize the camera pose by minimizing the reprojection error, thereby improving the accuracy of the camera pose.

[0066] S130 . Determine, according to the camera posture of the camera and each first feature point, a sparse point cloud corresponding to each first feature point.

[0067] The sparse point cloud is a point cloud generated by triangulating each first feature point.

[0068] Specifically, for each camera, the camera pose calculated in the above manner can be used to generate a corresponding sparse point cloud by triangulating each first feature point.

[0069] S140 , determining a three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera, and voxelizing the three-dimensional Gaussian distribution to determine each voxel and an occupancy value corresponding to each voxel.

[0070] The three-dimensional Gaussian distribution is a Gaussian distribution obtained by processing each point cloud point in each sparse point cloud with a three-dimensional Gaussian distribution and integrating them together, which may be an ellipsoidal Gaussian distribution. The occupancy value is a value used to describe the occupancy of the voxel.

[0071] Specifically, each point cloud point in each sparse point cloud corresponding to each camera is processed with a three-dimensional Gaussian, that is, each point cloud point is constructed into a three-dimensional Gaussian ellipsoid Gaussian distribution, and each ellipsoid point cloud is combined accordingly to obtain the three-dimensional Gaussian distribution corresponding to each camera as a whole. The three-dimensional Gaussian distribution is voxelized to obtain each voxel, and the occupancy value corresponding to each voxel can be obtained according to the distribution of the three-dimensional Gaussian distribution in each voxel.

[0072] S150 , determining a parking space detection frame corresponding to the key frame group according to the occupancy value of each voxel and a pre-trained three-dimensional point cloud parking space detection model.

[0073] The 3D point cloud parking space detection model is a pre-trained detection model for identifying parking spaces based on point clouds, and may be a voxelnet network (a deep learning network for 3D object detection based on point clouds) model for parking space recognition. The parking space detection box is a 3D detection box used to represent a parking space.

[0074] Specifically, according to the occupancy value of each voxel, the occupancy status of each voxel can be determined, that is, whether the voxel is occupied. Then, the occupancy status of each voxel is input into a pre-trained three-dimensional point cloud parking space detection model. Through processing by the model, the parking space detection frame corresponding to the key frame group can be obtained.

[0075] The parking space recognition method provided in the embodiment of the present application obtains each key frame group in the video frame captured by the camera for each camera, and determines each first feature point corresponding to the key frame group and the second feature point corresponding to each first feature point for each key frame group, and then determines the camera pose of the camera according to each first feature point and the second feature point corresponding to each first feature point, so as to facilitate subsequent conversion in three-dimensional space, and further, determines the sparse point cloud corresponding to each first feature point according to the camera pose of the camera and each first feature point, and determines the three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera, so as to convert The planar image is converted into three-dimensional space, and further, the three-dimensional Gaussian distribution is voxelized to determine each voxel and the occupancy value corresponding to each voxel. According to the occupancy value of each voxel and the pre-trained three-dimensional point cloud parking space detection model, the parking space detection frame corresponding to the key frame group is determined, thereby realizing the effect of identifying parking spaces in three-dimensional space and improving the recognition accuracy of non-planar parking spaces. Compared with the prior art, the parking space recognition method in which each camera is on the same plane by default takes non-planar scenes into consideration, and can effectively identify parking spaces in three-dimensional space. It can also accurately identify parking spaces in scenes with large slopes, thereby improving the subsequent parking experience.

[0076] Figure 2 is a flow chart of another parking space recognition method provided by an embodiment of the present application. Based on the above embodiments, the determination method of the three-dimensional Gaussian distribution and the calculation method of the occupancy value corresponding to each voxel are exemplarily described. Figure 2 , the parking space recognition method specifically includes:

[0077] S210: For each camera, obtain each key frame group in the video frame captured by the camera, and for each key frame group, determine each first feature point corresponding to the key frame group and a second feature point corresponding to each first feature point.

[0078] S220: Determine the camera pose of the camera according to each first feature point and the second feature point corresponding to each first feature point.

[0079] S230: Determine, according to the camera posture of the camera and each first feature point, a sparse point cloud corresponding to each first feature point.

[0080] S240 , according to the initial parameters, convert each point cloud point in each sparse point cloud corresponding to each camera into a single-point Gaussian distribution, and determine an initial Gaussian distribution according to each single-point Gaussian distribution.

[0081] The initial parameters are initialization parameters corresponding to the three-dimensional Gaussian distribution processing, including ellipsoid shape, color, opacity, etc. The single-point Gaussian distribution is the Gaussian distribution obtained by processing each point cloud point with a three-dimensional Gaussian distribution. The initial Gaussian distribution is the Gaussian distribution obtained by combining the single-point Gaussian distributions.

[0082] Specifically, the positions of each point cloud point in each sparse point cloud corresponding to each camera are combined to construct a point cloud set corresponding to each camera as a whole, and each point cloud point is converted into a corresponding single-point Gaussian distribution using the initial parameters. Then, these single-point Gaussian distributions are superimposed and combined to obtain the initial Gaussian distribution.

[0083] S250 , determining a rasterized image corresponding to each camera according to the initial Gaussian distribution and the camera posture of each camera.

[0084] The rasterized image is an image obtained by converting the initial Gaussian distribution according to the camera pose using a differentiable Gaussian rasterization technique.

[0085] Specifically, for each camera, the initial Gaussian distribution is transformed according to the camera pose corresponding to the camera using the differentiable Gaussian rasterization technology, and projected onto a two-dimensional image, so as to obtain a rasterized image corresponding to the camera.

[0086] S260. According to the rasterized images corresponding to the cameras and the key frames corresponding to the rasterized images, based on a preset loss function, each single-point Gaussian distribution is updated, and a three-dimensional Gaussian distribution is determined according to the updated single-point Gaussian distributions.

[0087] The preset loss function is a function used to evaluate the error between a rasterized image and a key frame corresponding to the rasterized image.

[0088] Specifically, for each camera, the rasterized image corresponding to the camera and the key frame corresponding to the rasterized image are input into the preset loss function to obtain the loss value. Taking the minimum sum of the loss values ​​corresponding to each camera as the goal, the parameters corresponding to each single-point Gaussian distribution are updated. Then, the updated single-point Gaussian distribution can be used to obtain a three-dimensional Gaussian distribution.

[0089] It can be understood that updating each single-point Gaussian distribution is actually updating the parameters used in the three-dimensional Gaussian distribution processing, that is, the updated single-point Gaussian distribution no longer uses the initial parameters, but uses the parameters after loss adjustment based on the preset loss function to make the three-dimensional Gaussian distribution closer to the real scene.

[0090] S270 , voxelizing the three-dimensional Gaussian distribution according to a preset resolution to obtain voxels.

[0091] The preset resolution is a preset resolution of each voxel grid, such as 0.02 m, etc. The specific value can be set according to the requirements.

[0092] Specifically, the three-dimensional Gaussian distribution is voxelized according to a preset resolution, and the three-dimensional Gaussian distribution is divided into voxels. It is understandable that each voxel may contain a Gaussian distribution or may not contain a Gaussian distribution.

[0093] S280 . For each voxel, determine the occupancy value of the voxel according to the updated single-point Gaussian distributions corresponding to the voxel.

[0094] Specifically, for each voxel, the coverage situation may be calculated according to each single-point Gaussian distribution contained in the voxel to obtain a corresponding occupancy value, so as to subsequently determine whether the voxel is occupied.

[0095] Based on the above example, the occupancy value of the voxel can be determined according to the updated single-point Gaussian distribution corresponding to the voxel based on the following formula:

[0096]

[0097] Among them, v j is the occupancy value of the jth voxel, N is the number of updated single-point Gaussian distributions corresponding to the voxel, α i is the opacity of the updated i-th single-point Gaussian distribution corresponding to the voxel, β i is the probability density of the updated i-th single-point Gaussian distribution corresponding to the voxel.

[0098] It is understandable that each voxel may be covered by multiple single-point Gaussian distributions, so it is necessary to calculate a single occupancy value for each single-point Gaussian distribution covering the voxel and sum them up as the occupancy value of the entire voxel.

[0099] S290 , determining a parking space detection frame corresponding to the key frame group according to the occupancy value of each voxel and a pre-trained three-dimensional point cloud parking space detection model.

[0100] Based on the above example, the parking space detection frame corresponding to the key frame group can be determined according to the occupancy value of each voxel and the pre-trained 3D point cloud parking space detection model in the following way:

[0101] For each voxel, in response to the occupancy value of the voxel being greater than a preset value, the occupancy value of the voxel is updated to 1; in response to the occupancy value of the voxel being less than or equal to the preset value, the occupancy value of the voxel is updated to 0;

[0102] The updated occupancy value of each voxel is input into the pre-trained 3D point cloud parking space detection model to obtain the parking space detection frame corresponding to the key frame group.

[0103] The preset value is a pre-set value used to determine whether a voxel is occupied, and the specific value can be determined according to demand.

[0104] Specifically, for each voxel, the size relationship between the occupancy value corresponding to the voxel and the preset value is determined. If the occupancy value of the voxel is greater than the preset value, the occupancy value of the voxel is updated to 1, that is, the voxel is determined to be occupied; if the occupancy value of the voxel is less than or equal to the preset value, the occupancy value of the voxel is updated to 0, that is, the voxel is determined to be unoccupied. The updated occupancy values ​​of each voxel are input into the pre-trained three-dimensional point cloud parking space detection model, and the parking space detection and recognition is performed through the three-dimensional point cloud parking space detection model. The output result of the model can be used as the parking space detection frame corresponding to the key frame group.

[0105] Figure 3 is a flowchart of a parking space identification method provided in an embodiment of the present application. Figure 3 As shown in the figure, first, multiple fisheye cameras are calibrated, and images are taken with the calibrated cameras. Multiple fisheye camera models are combined to form a camera model, and the camera model is used to perform real-time 3D reconstruction of the surrounding scene during parking. Among them, it mainly includes two parts: pose estimation and point cloud rendering. The sparse reconstruction in 3D reconstruction mainly uses the ORB descriptor to match feature points. At the same time, the motor speed and IMU on the vehicle are integrated in the pose estimation stage to improve the accuracy of pose estimation. In dense reconstruction, three-dimensional Gaussian splashing is mainly used for point cloud densification rendering. Finally, a 3D semantic detection model (three-dimensional point cloud parking space detection model) is constructed to realize the detection of 3D parking space frame (parking space detection frame). The above process combines the contents of camera model construction, pose estimation, point cloud rendering and semantic information recognition, providing reliable technical support for the accurate detection of three-dimensional parking space frame.

[0106] Multi-camera model construction: Considering that multiple (such as 4) fisheye cameras will be used in the parking process, a virtual camera model is constructed. The virtual camera model is located in the vehicle body coordinate system. The virtual camera model can be used to realize the conversion from the world coordinate system to the local camera coordinate system.

[0107] Specifically, assume that the three-dimensional point in the world coordinate system is represented by P i , the corresponding two-dimensional point in the video frame of the c-th camera at time t can be expressed as m itc , so the three-dimensional point P i and the two-dimensional point m itc The mapping relationship between them can be expressed as follows:

[0108]

[0109] in, Represents the homogeneous transformation matrix from the world coordinate system to the vehicle coordinate system, Represents the transformation matrix from the vehicle coordinate system to the local camera model, that is, the transformation matrix from the virtual camera model to the local camera model. The parameters of the transformation matrix can be obtained through the external parameter calibration of the fisheye camera. c It represents the local fisheye camera model, which can be obtained through the internal calibration of the fisheye camera and the distortion coefficient table provided by the camera manufacturer.

[0110] Pose estimation: In order to accurately estimate the camera pose for subsequent point cloud rendering, key frames will be selected for the video frames captured by the camera during the pose estimation process, that is, only key frames will be estimated for pose estimation. The principle of selecting key frames is that the feature matching point pair is less than the threshold T and the time is greater than 0.1s. The pose between key frames can be estimated as follows: Assume that the image corresponding to the tth time is The image corresponding to the t+1th moment is Using ORB feature point pair and The two groups of images (key frame groups) are respectively subjected to feature point extraction, and the extracted feature points are represented as m itc (i=1,…,M) and m it+1c (i=1,…,M), that is, each first feature point and the second feature point corresponding to each first feature point. Therefore, the SFM can be initialized by these two sets of matching points to obtain the initial relative pose between the two key frames. It is also possible to obtain the three-dimensional point P corresponding to each set of matching feature points i , and by fusing the extended Kalman filter algorithm with the motor speed and IMU, the relative pose (Kalman relative pose) between the two key frames can be obtained as Through and Nonlinear optimization can be performed to obtain the accurate camera pose between two key frames, denoted as Assuming that the position coordinates of the first frame in the key frame are the origin of the world coordinate system, we can continuously iterate and calculate the pose estimate of each key frame. In order to obtain a more accurate camera pose, after completing the pose estimate of each N frames, we need to perform bundle adjustment optimization, that is, optimize the camera pose by minimizing the reprojection error. The specific formula has been explained in the above example and will not be repeated here.

[0111] Point Cloud Rendering:

[0112] 1. Initialization: The camera pose calculated by the pose estimation module can triangulate the feature points in the keyframe to generate a sparse point cloud, and then each point in the sparse point cloud can be converted into a 3D Gaussian distribution (single-point Gaussian distribution). Each 3D Gaussian distribution S includes parameters such as the position μ (feature point position) of the center point, the shape Σ of the ellipsoid, the color (spherical harmonic function SH), and the opacity α. The shape, color, and opacity of the ellipsoid are obtained by random initialization (initial parameters). Then, the initial Gaussian distribution can be obtained.

[0113] 2. Rasterized image: The initial Gaussian distribution is converted into a rasterized image using a differentiable Gaussian rasterization technique. Specifically, it is assumed that the camera pose calculated in the pose estimation step can be represented by F 1c F 2c . The projection to the 2D image can be expressed as: Σ c ′ =J(F 1c F 2c ) -1 ΣF 1c F 2c -T J T . Where J represents the Jacobian matrix of the affine estimate of the projection function, Σ is the three-dimensional covariance matrix in the world coordinate system, Σ c ′ is the covariance matrix in the camera coordinate system of camera c. Along the projection matrix F 1c F 2c The corresponding rasterized image can be obtained by projecting it onto the camera plane. The RGB value of each pixel in the rasterized image can be determined by the parameters of the corresponding single-point Gaussian distribution (which can be N single-point Gaussian distributions). The RGB value is expressed as Among them, C i Obtained from the spherical harmonic function SH, α i Then through Σ c ′And the opacity α is multiplied to obtain the other single-point Gaussian distributions passed by the ray projected from the i-th single-point Gaussian distribution to the pixel point in the rasterized image, which is recorded as j.

[0114] 3. Densified rendering: By designing a preset loss function between the rasterized image and the real image (taken by the camera), and minimizing the function value of the preset loss function, the parameters such as the ellipsoid shape Σ, color (spherical harmonic function SH), and opacity α can be optimized. The preset loss function is defined as Among them, RGB m Represents the RGB value of the mth point in the rasterized image, RGB gt represents the RGB value of the mth point in the input image (key frame). By minimizing L ct (Multi-channel camera L ct The point cloud is rendered densely by performing multiple back propagations to obtain a three-dimensional Gaussian distribution.

[0115] Parking space frame detection and occupied space detection: After dense rendering, the real-time reconstructed parking scene will be subjected to 3D parking space frame detection. Current 3D object detection methods are applicable to point cloud detection and voxelization methods, which are difficult to apply to 3D Gaussian modules. Therefore, a pre-processing module is added to voxelnet (a pre-trained 3D point cloud parking space detection model), which can voxelize the input 3D Gaussian distribution to be sent to voxelnet for 3D parking space frame detection. For example, a voxel grid of X×Y×Z can be constructed by the mean μ in the 3D Gaussian distribution, where the resolution of the voxel grid is a preset size, such as 0.02m. Therefore, for each voxel in X×Y×Z, the probability density and opacity of the single-point Gaussian distribution corresponding to the voxel are used to determine the occupancy value. The specific formula has been explained in the above example and will not be repeated here. Therefore, through the above pre-processing modules, the three-dimensional Gaussian distribution can be voxelized. Finally, 3D parking space detection can be performed through voxelnet to obtain a 3D parking space detection frame (parking space detection frame), which can be applied to other downstream tasks in the parking process.

[0116] The parking space recognition method provided in the embodiment of the present application converts each point cloud point in each sparse point cloud corresponding to each camera into a single-point Gaussian distribution according to initial parameters, determines the initial Gaussian distribution according to each single-point Gaussian distribution, determines the rasterized image corresponding to each camera according to the initial Gaussian distribution and the camera posture of each camera, so as to project the initial Gaussian distribution onto the camera screen, updates each single-point Gaussian distribution according to the rasterized image corresponding to each camera and the key frame corresponding to the rasterized image based on a preset loss function, improves the matching degree between the single-point Gaussian distribution and the actual scene, determines a three-dimensional Gaussian distribution according to the updated single-point Gaussian distribution, voxelizes the three-dimensional Gaussian distribution according to a preset resolution, obtains each voxel, and provides a suitable input for the three-dimensional point cloud parking space detection model, thereby realizing the construction of a three-dimensional Gaussian distribution matching the actual scene, improving the accuracy of point cloud reconstruction, and improving the accuracy of subsequent parking space detection.

[0117] Figure 4 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 400 includes one or more processors 401 and a memory 402 .

[0118] The processor 401 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0119] The memory 402 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may run the program instructions to implement the parking space identification method of any embodiment of the present application described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0120] In one example, the electronic device 400 may further include: an input device 403 and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 403 may include, for example, a keyboard, a mouse, etc. The output device 404 may output various information to the outside, including early warning information, braking force, etc. The output device 404 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0121] Of course, to simplify, Figure 4 Only some of the components related to the present application in the electronic device 400 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 400 may also include any other appropriate components.

[0122] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the parking space identification method provided by any embodiment of the present application.

[0123] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0124] In addition, the embodiments of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the parking space identification method provided by any embodiment of the present application.

[0125] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0126] It should be noted that the terms used in this application are only for describing specific embodiments, rather than limiting the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0127] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0128] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and its core ideas of this application. The above is only the preferred implementation method of this application. It should be pointed out that due to the limitations of textual expression and the objective existence of infinite specific structures, ordinary technicians in this technical field can make several improvements, modifications or changes without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the inventive concept and technical solution to other occasions without improvement, should be regarded as the scope of protection of this application.

Claims

1. A parking space recognition method, characterized in that: include: For each camera, each key frame group in the video frame captured by the camera is obtained, and for each key frame group, each first feature point corresponding to the key frame group and a second feature point corresponding to each first feature point are determined; wherein the key frame group is a combination of every two adjacent key frames in the video frame; Determining a camera pose of the camera according to each first feature point and a second feature point corresponding to each first feature point; Determine, according to the camera pose of the camera and each first feature point, a sparse point cloud corresponding to each first feature point; Determine a three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera, and voxelize the three-dimensional Gaussian distribution to determine each voxel and an occupancy value corresponding to each voxel; According to the occupancy value of each voxel and the pre-trained three-dimensional point cloud parking space detection model, the parking space detection frame corresponding to the key frame group is determined.

2. The method according to claim 1, characterized in that The step of determining the camera pose of the camera according to each first feature point and a second feature point corresponding to each first feature point includes: Determine an initial relative pose based on the first feature points and the second feature points corresponding to the first feature points and the structure from motion algorithm; Acquire motor speed and inertial measurement information, and determine Kalman relative posture according to the motor speed and the inertial measurement information; The camera pose of the camera is determined according to the initial relative pose and the Kalman relative pose.

3. The method according to claim 2, characterized in that After determining the camera pose of the camera according to the initial relative pose and the Kalman relative pose, the method further includes: According to the feature points in the key frames of each camera at consecutive preset moments, and the three-dimensional points corresponding to each feature point, the preset objective function is minimized and the camera posture of each camera is updated; wherein the preset objective function is constructed based on the error between the conversion points corresponding to each three-dimensional point observed by each camera and the feature points in the key frames of each camera at each preset moment.

4. The method according to claim 3, characterized in that The preset objective function is: Among them, P i is the i-th 3D point among the 3D points observed by all cameras, N is the total number of 3D points, M is the total number at the preset time, C is the total number of cameras, is the effective value of the i-th 3D point observed by the c-th camera at the t-th time, m itc is the feature point corresponding to the i-th 3D point of the c-th camera at the t-th time, ||·|| huber is the smooth L1 norm, is the preset external parameter transformation matrix of the c-th camera, is the homogeneous transformation matrix to be solved, π c is the local camera model of the c-th camera, is the transformation point corresponding to the i-th 3D point, F 1c F 2c is the camera pose of the c-th camera.

5. The method according to claim 1, characterized in that Determining the three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera includes: According to the initial parameters, each point cloud point in each sparse point cloud corresponding to each camera is converted into a single-point Gaussian distribution, and the initial Gaussian distribution is determined according to each single-point Gaussian distribution; Determine a rasterized image corresponding to each camera according to the initial Gaussian distribution and the camera pose of each camera; According to the rasterized images corresponding to the cameras and the key frames corresponding to the rasterized images, each single-point Gaussian distribution is updated based on a preset loss function, and a three-dimensional Gaussian distribution is determined according to the updated single-point Gaussian distributions.

6. The method according to claim 5, characterized in that The voxelization of the three-dimensional Gaussian distribution to determine each voxel and an occupancy value corresponding to each voxel includes: According to a preset resolution, the three-dimensional Gaussian distribution is voxelized to obtain each voxel; For each voxel, the occupancy value of the voxel is determined according to each updated single-point Gaussian distribution corresponding to the voxel.

7. The method according to claim 6, characterized in that The determining the occupancy value of the voxel according to each updated single-point Gaussian distribution corresponding to the voxel includes: The occupancy value of the voxel is determined based on the following formula: Among them, v j is the occupancy value of the jth voxel, N is the number of updated single-point Gaussian distributions corresponding to the voxel, α i is the opacity of the updated i-th single-point Gaussian distribution corresponding to the voxel, β i is the probability density of the updated i-th single-point Gaussian distribution corresponding to the voxel.

8. The method according to claim 1, characterized in that Determining the parking space detection frame corresponding to the key frame group according to the occupancy value of each voxel and the pre-trained three-dimensional point cloud parking space detection model includes: For each voxel, in response to the occupancy value of the voxel being greater than a preset value, the occupancy value of the voxel is updated to 1; in response to the occupancy value of the voxel being less than or equal to the preset value, the occupancy value of the voxel is updated to 0; The updated occupancy value of each voxel is input into a pre-trained three-dimensional point cloud parking space detection model to obtain a parking space detection frame corresponding to the key frame group.

9. An electronic device, characterized in that: The electronic device comprises: Processor and memory; The processor is used to execute the steps of the parking space recognition method according to any one of claims 1 to 8 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the parking space recognition method according to any one of claims 1 to 8.

Images