Parking space identification method, electronic device and medium

Abstract

The present application provides a parking space recognition method, electronic device and medium, which include: for each camera, obtaining each key frame group in the video frame captured by the camera, and for each key frame group, determining each first feature point corresponding to the key frame group and the second feature point corresponding to each first feature point; determining the camera posture of the camera according to each first feature point and the second feature point corresponding to each first feature point; determining the sparse point cloud corresponding to each first feature point according to the camera posture of the camera and each first feature point; determining a three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera, voxelizing the three-dimensional Gaussian distribution, and determining each voxel and the occupancy value corresponding to each voxel; determining the parking space detection frame corresponding to the key frame group according to the occupancy value of each voxel and a pre-trained three-dimensional point cloud parking space detection model, so as to realize parking space recognition in three-dimensional space and improve the recognition accuracy of non-planar parking spaces.

Description

Technical Field

[0001] The present application relates to the field of assisted driving technology, and in particular to a parking space recognition method, electronic equipment, and medium. Background Art

[0002] Parking space perception is a key function of automated parking assistance systems. Automatic parking typically involves finding a suitable parking space within a parking lot, executing precise parking maneuvers, and avoiding collisions with surrounding obstacles. Parking space perception plays a crucial role in this process. Traditional parking space perception algorithms typically use a fisheye camera to construct an AVM (Around View Monitor) image, then perform parking space frame perception on this image.

[0003] However, this approach typically assumes that the scene captured by the four cameras is generally flat. When a vehicle encounters a steep slope, the surround view images stitched together by the four fisheye cameras can become misaligned, leading to significant discrepancies between the identified parking spaces and the actual space, hindering the automated parking process. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, the present application aims to provide a parking space recognition method, electronic equipment and medium to realize the recognition of parking spaces in three-dimensional space and improve the recognition accuracy of non-planar parking spaces.

[0005] The present invention provides a method for identifying a parking space, the method comprising:

[0006] For each camera, obtain each key frame group in the video frame captured by the camera, and for each key frame group, determine each first feature point corresponding to the key frame group and each second feature point corresponding to the first feature point; wherein the key frame group is a combination of every two adjacent key frames in the video frame;

[0007] Determining a camera pose of the camera according to each first feature point and a second feature point corresponding to each first feature point;

[0008] Determining, according to the camera pose of the camera and the first feature points, a sparse point cloud corresponding to each first feature point;

[0009] Determine a three-dimensional Gaussian distribution based on each point cloud point in each sparse point cloud corresponding to each camera, and voxelize the three-dimensional Gaussian distribution to determine each voxel and its corresponding occupancy value;

[0010] The parking space detection frame corresponding to the key frame group is determined according to the occupancy value of each voxel and a pre-trained three-dimensional point cloud parking space detection model.

[0011] According to the technical solution provided in the embodiment of the present application, optionally, determining the camera pose of the camera according to each first feature point and the second feature point corresponding to each first feature point includes:

[0012] Determining an initial relative pose based on a structure-from-motion algorithm according to each first feature point and a second feature point corresponding to each first feature point;

[0013] Obtaining motor speed and inertial measurement information, and determining a Kalman relative posture based on the motor speed and the inertial measurement information;

[0014] The camera pose of the camera is determined according to the initial relative pose and the Kalman relative pose.

[0015] According to the technical solution provided in the embodiment of the present application, optionally, after determining the camera pose of the camera according to the initial relative pose and the Kalman relative pose, the method further includes:

[0016] Based on the feature points in the key frames of each camera at consecutive preset moments and the three-dimensional points corresponding to each feature point, the preset objective function is minimized and the camera pose of each camera is updated; wherein the preset objective function is constructed based on the error between the conversion points corresponding to each three-dimensional point commonly observed by each camera and the feature points in the key frames of each camera at each preset moment.

[0017] According to the technical solution provided in the embodiment of the present application, optionally, the preset objective function is:

[0018]

[0019] Among them, P i is the i-th 3D point among all 3D points observed by all cameras, N is the total number of 3D points, M is the total number at the preset time, and C is the total number of cameras. is the effective value of the i-th 3D point observed by the c-th camera at the t-th moment, m itc is the feature point corresponding to the i-th 3D point of the c-th camera at the t-th moment, ||·|| huber is the smooth L1 norm, is the preset external parameter transformation matrix of the c-th camera, is the homogeneous transformation matrix to be solved, π c is the local camera model of the c-th camera, is the transformation point corresponding to the i-th 3D point, F 1c F 2c is the camera pose of the c-th camera.

[0020] According to the technical solution provided in the embodiment of the present application, optionally, determining the three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera includes:

[0021] According to the initial parameters, each point cloud point in each sparse point cloud corresponding to each camera is converted into a single-point Gaussian distribution, and the initial Gaussian distribution is determined based on each single-point Gaussian distribution;

[0022] Determining a rasterized image corresponding to each camera according to the initial Gaussian distribution and the camera pose of each camera;

[0023] According to the rasterized images corresponding to the cameras and the key frames corresponding to the rasterized images, each single-point Gaussian distribution is updated based on a preset loss function, and a three-dimensional Gaussian distribution is determined based on the updated single-point Gaussian distribution.

[0024] According to the technical solution provided in the embodiment of the present application, optionally, performing voxel processing on the three-dimensional Gaussian distribution to determine each voxel and the occupancy value corresponding to each voxel includes:

[0025] voxelize the three-dimensional Gaussian distribution according to a preset resolution to obtain voxels;

[0026] For each voxel, the occupancy value of the voxel is determined according to the updated single-point Gaussian distributions corresponding to the voxel.

[0027] According to the technical solution provided in the embodiment of the present application, optionally, determining the occupancy value of the voxel according to each updated single-point Gaussian distribution corresponding to the voxel includes:

[0028] The occupancy value of the voxel is determined based on the following formula:

[0029]

[0030] Among them, v j is the occupancy value of the j-th voxel, N is the number of updated single-point Gaussian distributions corresponding to the voxel, α i is the opacity of the updated i-th single-point Gaussian distribution corresponding to the voxel, β i is the probability density of the updated i-th single-point Gaussian distribution corresponding to the voxel.

[0031] According to the technical solution provided in the embodiment of the present application, optionally, determining the parking space detection frame corresponding to the key frame group based on the occupancy value of each voxel and a pre-trained three-dimensional point cloud parking space detection model includes:

[0032] For each voxel, in response to the occupancy value of the voxel being greater than a preset value, the occupancy value of the voxel is updated to 1; in response to the occupancy value of the voxel being less than or equal to the preset value, the occupancy value of the voxel is updated to 0;

[0033] The updated occupancy value of each voxel is input into a pre-trained three-dimensional point cloud parking space detection model to obtain a parking space detection frame corresponding to the key frame group.

[0034] An embodiment of the present application further provides an electronic device, comprising:

[0035] processor and memory;

[0036] The processor is configured to execute the steps of the parking space recognition method as described in any embodiment by calling the program or instructions stored in the memory.

[0037] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute the steps of the parking space identification method as described in any embodiment.

[0038] In summary, the present application proposes a parking space recognition method, which obtains each key frame group in the video frame captured by the camera for each camera, determines each first feature point corresponding to the key frame group and the second feature point corresponding to each first feature point for each key frame group, and then determines the camera pose of the camera according to each first feature point and the second feature point corresponding to each first feature point, so as to facilitate subsequent conversion in three-dimensional space, further, according to the camera pose of the camera and each first feature point, determines the sparse point cloud corresponding to each first feature point, and determines the three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera, so as to facilitate subsequent conversion in three-dimensional space, further, according to the camera pose of the camera and each first feature point, determines the sparse point cloud corresponding to each first feature point, and determines the three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera, so as to facilitate subsequent conversion in three-dimensional space, The planar image is converted into three-dimensional space. Furthermore, the three-dimensional Gaussian distribution is voxelized to determine each voxel and the occupancy value corresponding to each voxel. According to the occupancy value of each voxel and the pre-trained three-dimensional point cloud parking space detection model, the parking space detection frame corresponding to the key frame group is determined, thereby realizing the effect of identifying parking spaces in three-dimensional space and improving the recognition accuracy of non-planar parking spaces. Compared with the existing technology, the parking space recognition method of each camera defaults to the same plane. Taking non-planar scenes into consideration, it can effectively identify parking spaces in three-dimensional space and accurately identify parking spaces in scenes with large slopes, thereby improving the subsequent parking experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of a parking space identification method provided by an embodiment of the present application;

[0040] Figure 2is a flow chart of another parking space identification method provided in an embodiment of the present application;

[0041] Figure 3 This is a flowchart of a parking space identification method according to an embodiment of the present application;

[0042] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the invention are shown in the accompanying drawings.

[0044] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] Figure 1 This is a flow chart of a parking space identification method provided by an embodiment of the present application. Figure 1 , the parking space recognition method specifically includes:

[0046] S110 : For each camera, obtain each key frame group in the video frame captured by the camera, and for each key frame group, determine each first feature point corresponding to the key frame group and each second feature point corresponding to the first feature point.

[0047] Among them, in order to perform parking space recognition in three-dimensional space, this embodiment requires the use of multiple cameras, such as four-way cameras, or more-way cameras, and the type of camera can be a fisheye camera. Video frames are continuous image frames captured by the camera. A key frame group is a combination of every two adjacent key frames in a video frame. A key frame is a video frame that meets the key frame requirements, and the key frame contains important states or key changes. The key frame requirement can be that the feature matching point pair with the adjacent key frame (each first feature point and the second feature point corresponding to each first feature point) is less than a threshold and the time is greater than a time threshold (such as 0.1s). The first feature point and the second feature point corresponding to the first feature point are feature points located in two different key frames in the key frame group, respectively, and have a matching relationship.

[0048] Specifically, for each camera, video frames are collected separately, key frames are extracted from the video frames, and the combination of every two adjacent key frames is regarded as a key frame group. In addition, the feature point matching pairs corresponding to each key frame group can be determined. For example, the ORB (Oriented FAST and Rotated BRIEF) algorithm can be used to extract each first feature point and the second feature point corresponding to each first feature point.

[0049] S120 : Determine a camera pose of the camera according to each first feature point and a second feature point corresponding to each first feature point.

[0050] Among them, the camera pose refers to the position and posture of the camera in the world coordinate system, which is usually determined by rotation and translation.

[0051] Specifically, the camera pose corresponding to each keyframe group can be estimated based on the position of each first feature point and the second feature point corresponding to each first feature point. It is understood that the first frame in each keyframe can be used as the origin of the world coordinate system, and subsequent solutions can be based on this.

[0052] Based on the above example, the camera pose can be estimated by combining the SFM (Structure from Motion) initialization method and the Extended Kalman Filter (EKF) algorithm. Specifically, the camera pose can be determined based on each first feature point and the second feature point corresponding to each first feature point in the following manner:

[0053] Determining an initial relative pose based on a structure-from-motion algorithm according to each first feature point and a second feature point corresponding to each first feature point;

[0054] Obtain motor speed and inertial measurement information, and determine the Kalman relative posture based on the motor speed and inertial measurement information;

[0055] The camera pose is determined based on the initial relative pose and the Kalman relative pose.

[0056] Among them, the motion recovery structure algorithm is a three-dimensional reconstruction method that estimates the geometric structure of the scene by obtaining corresponding points from multiple images. The initial relative pose is the result of the motion recovery structure algorithm solving the camera pose. The motor speed is the number of revolutions per minute of the motor, which can be collected by the corresponding sensor. The inertial measurement information is based on the information collected by the inertial measurement unit (IMU), such as acceleration, angular velocity, and direction. The Kalman relative pose is the result of solving the camera pose by the extended Kalman filter algorithm.

[0057] Specifically, the structure-from-motion algorithm is used to calculate the camera pose for each first feature point and its corresponding second feature point, obtaining the initial relative pose. The EKF algorithm is used to determine the relative pose of two keyframes in a keyframe group, known as the Kalman relative pose. Nonlinear optimization is performed on the initial relative pose and the Kalman relative pose to obtain the camera pose.

[0058] Based on the above example, after determining the camera pose based on the initial relative pose and the Kalman relative pose, it can also be optimized through multi-frame bundling adjustment, specifically:

[0059] Based on the feature points in the key frames of each camera at consecutive preset moments and the 3D points corresponding to each feature point, the preset objective function is minimized and the camera pose of each camera is updated.

[0060] The preset moments are multiple consecutive moments, such as 50 consecutive moments, and the specific number can be set as needed. The 3D points are points in the 3D space captured by each camera, and correspond to feature points in the keyframes. This correspondence can be derived using a structure-from-motion algorithm. The preset objective function is constructed based on the error between the transformation points corresponding to each 3D point observed by each camera and each feature point in the keyframes of each camera at each preset moment.

[0061] Specifically, each feature point in the key frame of each camera at each consecutive preset moment, as well as the three-dimensional point corresponding to each feature point, is input into a preset objective function to obtain the objective function value. With the minimum objective function value as the goal, the camera pose of each camera is solved, and the camera pose of each camera is updated based on the camera pose obtained by the optimization solution.

[0062] Based on the above example, the preset objective function can be:

[0063]

[0064] Among them, P iis the i-th 3D point among all 3D points observed by all cameras, N is the total number of 3D points, M is the total number at the preset time, and C is the total number of cameras. is the effective value of the i-th 3D point observed by the c-th camera at the t-th moment, m itc is the feature point corresponding to the i-th 3D point of the c-th camera at the t-th moment, ||·|| huber is the smooth L1 norm, is the preset external parameter transformation matrix of the c-th camera, is the homogeneous transformation matrix to be solved, π c is the local camera model of the c-th camera, is the transformation point corresponding to the i-th 3D point, F 1c F 2c is the camera pose of the c-th camera.

[0065] It can be understood that optimizing the camera pose using the preset objective function is to optimize the camera pose by minimizing the reprojection error, thereby improving the accuracy of the camera pose.

[0066] S130 : Determine, based on the camera pose and the first feature points, a sparse point cloud corresponding to each first feature point.

[0067] The sparse point cloud is a point cloud generated by triangulating each first feature point.

[0068] Specifically, for each camera, the camera pose calculated in the above manner can be used to generate a corresponding sparse point cloud by triangulating each first feature point.

[0069] S140 , determining a three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera, and voxelizing the three-dimensional Gaussian distribution to determine each voxel and an occupancy value corresponding to each voxel.

[0070] The three-dimensional Gaussian distribution is a Gaussian distribution obtained by performing three-dimensional Gaussian distribution processing on each point in each sparse point cloud and integrating the obtained Gaussian distribution. The Gaussian distribution may be an ellipsoidal Gaussian distribution. The occupancy value is a numerical value used to describe the occupancy of the voxel.

[0071] Specifically, each point in the sparse point cloud corresponding to each camera is subjected to 3D Gaussian processing. This constructs each point into a 3D Gaussian ellipsoidal Gaussian distribution. The ellipsoidal point clouds are then combined to obtain the 3D Gaussian distribution corresponding to each camera. The 3D Gaussian distribution is then voxelized to obtain voxels. Based on the distribution of the 3D Gaussian distribution within each voxel, the occupancy value of each voxel can be determined.

[0072] S150 : Determine a parking space detection frame corresponding to the key frame group according to the occupancy value of each voxel and a pre-trained three-dimensional point cloud parking space detection model.

[0073] The 3D point cloud parking space detection model is a pre-trained detection model for identifying parking spaces based on point clouds, and can be a VoxelNet network (a deep learning network for 3D object detection based on point clouds) model for parking space recognition. The parking space detection box is a 3D detection box used to represent a parking space.

[0074] Specifically, based on the occupancy value of each voxel, the occupancy status of each voxel can be determined, that is, whether the voxel is occupied. Then, the occupancy status of each voxel is input into a pre-trained three-dimensional point cloud parking space detection model. After processing through the model, the parking space detection frame corresponding to the key frame group can be obtained.

[0075] The parking space recognition method provided in the embodiment of the present application obtains each key frame group in the video frame captured by the camera for each camera, determines each first feature point corresponding to the key frame group and the second feature point corresponding to each first feature point for each key frame group, and then determines the camera pose of the camera according to each first feature point and the second feature point corresponding to each first feature point, so as to facilitate subsequent conversion in three-dimensional space, further, according to the camera pose of the camera and each first feature point, determines the sparse point cloud corresponding to each first feature point, and determines the three-dimensional Gaussian distribution according to each point cloud point in each sparse point cloud corresponding to each camera, so as to convert The planar image is converted into three-dimensional space, and the three-dimensional Gaussian distribution is further voxelized to determine each voxel and the occupancy value corresponding to each voxel. According to the occupancy value of each voxel and the pre-trained three-dimensional point cloud parking space detection model, the parking space detection frame corresponding to the key frame group is determined, thereby realizing the effect of identifying parking spaces in three-dimensional space and improving the recognition accuracy of non-planar parking spaces. Compared with the existing technology, the parking space recognition method in which each camera defaults to the same plane is used. Taking non-planar scenes into consideration, it can effectively identify parking spaces in three-dimensional space and accurately identify parking spaces in scenes with large slopes, thereby improving the subsequent parking experience.

[0076] Figure 2 This is a flow chart of another parking space recognition method provided by an embodiment of the present application. Based on the above embodiments, the determination method of the three-dimensional Gaussian distribution and the calculation method of the occupancy value corresponding to each voxel are exemplarily described. Figure 2 , the parking space recognition method specifically includes:

[0077] S210 : For each camera, obtain each key frame group in the video frame captured by the camera, and for each key frame group, determine each first feature point corresponding to the key frame group and each second feature point corresponding to the first feature point.

[0078] S220 : Determine a camera pose of the camera according to each first feature point and a second feature point corresponding to each first feature point.

[0079] S230 : Determine, based on the camera pose and the first feature points, a sparse point cloud corresponding to each first feature point.

[0080] S240 , converting each point cloud point in each sparse point cloud corresponding to each camera into a single-point Gaussian distribution according to the initial parameters, and determining an initial Gaussian distribution according to each single-point Gaussian distribution.

[0081] The initial parameters are the initialization parameters corresponding to the 3D Gaussian distribution, including the ellipsoid shape, color, and opacity. The single-point Gaussian distribution is the Gaussian distribution obtained by applying the 3D Gaussian distribution to each point in the point cloud. The initial Gaussian distribution is the Gaussian distribution obtained by combining the single-point Gaussian distributions.

[0082] Specifically, the positions of each point in the sparse point cloud corresponding to each camera are combined to construct a point cloud set corresponding to each camera. Using the initial parameters, each point in the point cloud is converted into a corresponding single-point Gaussian distribution. These single-point Gaussian distributions are then superimposed and combined to obtain the initial Gaussian distribution.

[0083] S250 : Determine a rasterized image corresponding to each camera according to the initial Gaussian distribution and the camera pose of each camera.

[0084] The rasterized image is an image obtained by converting the initial Gaussian distribution according to the camera pose using the differentiable Gaussian rasterization technology.

[0085] Specifically, for each camera, the initial Gaussian distribution is transformed according to the camera pose corresponding to the camera using the differentiable Gaussian rasterization technology, and projected onto a two-dimensional image to obtain a rasterized image corresponding to the camera.

[0086] S260. According to the rasterized image corresponding to each camera and the key frame corresponding to the rasterized image, based on a preset loss function, update each single-point Gaussian distribution, and determine a three-dimensional Gaussian distribution according to the updated single-point Gaussian distribution.

[0087] The preset loss function is a function used to evaluate the error between the rasterized image and the key frame corresponding to the rasterized image.

[0088] Specifically, for each camera, the rasterized image corresponding to the camera and the key frame corresponding to the rasterized image are input into the preset loss function to obtain the loss value. With the goal of minimizing the sum of the loss values ​​corresponding to each camera, the parameters corresponding to each single-point Gaussian distribution are updated. Then, the updated single-point Gaussian distribution can be used to obtain a three-dimensional Gaussian distribution.

[0089] It can be understood that updating each single-point Gaussian distribution is actually updating the parameters used in the three-dimensional Gaussian distribution processing. That is, the updated single-point Gaussian distribution no longer uses the initial parameters, but uses the parameters after loss adjustment based on the preset loss function to make the three-dimensional Gaussian distribution closer to the real scene.

[0090] S270 , voxelizing the three-dimensional Gaussian distribution according to a preset resolution to obtain voxels.

[0091] The preset resolution is a preset resolution of each voxel grid, such as 0.02m, etc. The specific value can be set according to needs.

[0092] Specifically, the three-dimensional Gaussian distribution is voxelized according to a preset resolution, and the three-dimensional Gaussian distribution is divided into voxels. It is understandable that each voxel may contain a Gaussian distribution or may not contain a Gaussian distribution.

[0093] S280 : For each voxel, determine the occupancy value of the voxel according to the updated single-point Gaussian distributions corresponding to the voxel.

[0094] Specifically, for each voxel, the coverage situation may be calculated according to each single-point Gaussian distribution contained in the voxel to obtain a corresponding occupancy value, so as to facilitate subsequent determination of whether the voxel is occupied.

[0095] Based on the above example, the occupancy value of a voxel can be determined based on the updated single-point Gaussian distribution corresponding to the voxel using the following formula:

[0096]

[0097] Among them, v j is the occupancy value of the j-th voxel, N is the number of updated single-point Gaussian distributions corresponding to the voxel, α i is the opacity of the updated i-th single-point Gaussian distribution corresponding to the voxel, β i is the probability density of the updated i-th single-point Gaussian distribution corresponding to the voxel.

[0098] It is understandable that each voxel may be covered by multiple single-point Gaussian distributions, so it is necessary to calculate a single occupancy value for each single-point Gaussian distribution covering the voxel and sum them up as the occupancy value of the entire voxel.

[0099] S290 , determining a parking space detection frame corresponding to the key frame group according to the occupancy value of each voxel and a pre-trained three-dimensional point cloud parking space detection model.

[0100] Based on the above example, the parking space detection box corresponding to the key frame group can be determined according to the occupancy value of each voxel and the pre-trained 3D point cloud parking space detection model in the following way:

[0101] For each voxel, in response to the occupancy value of the voxel being greater than a preset value, the occupancy value of the voxel is updated to 1; in response to the occupancy value of the voxel being less than or equal to the preset value, the occupancy value of the voxel is updated to 0;

[0102] The updated occupancy value of each voxel is input into the pre-trained 3D point cloud parking space detection model to obtain the parking space detection box corresponding to the key frame group.

[0103] The preset value is a pre-set value used to determine whether a voxel is occupied, and the specific value can be determined according to needs.

[0104] Specifically, for each voxel, the relationship between the occupancy value corresponding to that voxel and a preset value is determined. If the voxel's occupancy value is greater than the preset value, the voxel's occupancy value is updated to 1, indicating that the voxel is occupied. If the voxel's occupancy value is less than or equal to the preset value, the voxel's occupancy value is updated to 0, indicating that the voxel is unoccupied. The updated occupancy values ​​of each voxel are input into a pre-trained 3D point cloud parking space detection model. Parking space detection and recognition are performed using the 3D point cloud parking space detection model, and the model's output can be used as the parking space detection box corresponding to the keyframe group.

[0105] Figure 3 This is a flowchart of a parking space identification method provided by an embodiment of the present application. Figure 3 As shown, first, multiple fisheye cameras are calibrated and images are captured using the calibrated cameras. The multiple fisheye camera models are combined to form a camera model, and this camera model is used to perform real-time 3D reconstruction of the surrounding scene during parking. This mainly includes two parts: pose estimation and point cloud rendering. Sparse reconstruction in 3D reconstruction mainly uses the ORB descriptor for feature point matching. At the same time, the vehicle-mounted motor speed and IMU are integrated in the pose estimation stage to improve the accuracy of pose estimation. In dense reconstruction, three-dimensional Gaussian splashing is mainly used for point cloud densification rendering. Finally, a 3D semantic detection model (three-dimensional point cloud parking space detection model) is constructed to realize the detection of 3D parking space frames (parking space detection frames). The above process combines the contents of camera model construction, pose estimation, point cloud rendering and semantic information recognition, providing reliable technical support for the accurate detection of three-dimensional parking space frames.

[0106] Multi-camera model construction: Considering that multiple (e.g., four) fisheye cameras will be used during the parking process, a virtual camera model is constructed. This virtual camera model is located in the vehicle coordinate system, and the conversion from the world coordinate system to the local camera coordinate system can be realized through this virtual camera model.

[0107] Specifically, assuming that the three-dimensional point in the world coordinate system is represented by P i , the corresponding two-dimensional point in the video frame of the c-th camera at time t can be expressed as m itc , so the three-dimensional point P i and the two-dimensional point m itc The mapping relationship between them can be expressed as follows:

[0108]

[0109] in, Represents the homogeneous transformation matrix from the world coordinate system to the vehicle coordinate system, Represents the transformation matrix from the vehicle coordinate system to the local camera model, that is, the transformation matrix from the virtual camera model to the local camera model. The parameters of the transformation matrix can be obtained through the external parameter calibration of the fisheye camera. c It represents the local fisheye camera model, which can be obtained through the intrinsic calibration of the fisheye camera and the distortion coefficient table provided by the camera manufacturer.

[0110] Pose estimation: In order to accurately estimate the camera pose for subsequent point cloud rendering, key frames will be selected from the video frames captured by the camera during the pose estimation process, that is, only key frames will be used for pose estimation. The principle of selecting key frames is that the feature matching point pair is less than the threshold T and the time is greater than 0.1s. The pose between key frames can be estimated as follows: Assume that the image corresponding to time t is The image corresponding to the t+1th moment is Using ORB feature point pairs and Feature points are extracted from two groups of images (key frame groups) respectively, and the extracted feature points are represented as m itc (i=1,…,M) and m it+1c (i=1,…,M), namely, each first feature point and the second feature point corresponding to each first feature point. Therefore, the SFM can be initialized by these two sets of matching points to obtain the initial relative pose between the two key frames. It is also possible to obtain the three-dimensional point P corresponding to each set of matching feature points i , and by fusing the extended Kalman filter algorithm of motor speed and IMU, the relative pose (Kalman relative pose) between the two key frames can be obtained as Through and The accurate camera pose between two key frames can be obtained by performing nonlinear optimization, which is recorded as Assuming the position coordinates of the first keyframe are the world coordinate system origin, we can iterate continuously to calculate the pose estimate for each keyframe. To obtain a more accurate camera pose, after completing the pose estimate for each N frames, we need to perform bundle adjustment optimization. This is to optimize the camera pose by minimizing the reprojection error. The specific formula is explained in the example above and will not be repeated here.

[0111] Point Cloud Rendering:

[0112] 1. Initialization: The camera pose calculated by the pose estimation module can be used to triangulate the feature points in the keyframe to generate a sparse point cloud. Furthermore, each point in the sparse point cloud can be converted into a 3D Gaussian distribution (single-point Gaussian distribution). Each 3D Gaussian distribution S includes parameters such as the center point position μ (feature point position), ellipsoid shape Σ, color (spherical harmonic function SH), and opacity α. The ellipsoid shape, color, and opacity are obtained through random initialization (initial parameters). Thus, the initial Gaussian distribution can be obtained.

[0113] 2. Rasterized image: The initial Gaussian distribution is converted into a rasterized image using the differentiable Gaussian rasterization technique. Specifically, it is assumed that the camera pose calculated in the pose estimation step can be represented by F 1c F 2c The projection to the 2D image can be expressed as: Σ c ′ =J(F 1c F 2c ) -1 ΣF 1c F 2c -T J T Where J represents the Jacobian matrix of the affine estimate of the projection function, Σ is the three-dimensional covariance matrix in the world coordinate system, Σ c ′ is the covariance matrix in the camera coordinate system of the c-way camera. Along the projection matrix F 1c F 2c The corresponding rasterized image can be obtained by projecting it onto the camera plane. The RGB value of each pixel in the rasterized image can be determined by the parameters of the corresponding single-point Gaussian distribution (which can be N single-point Gaussian distributions). The RGB value is expressed as Among them, C i Obtained from the spherical harmonic function SH, α i Then through Σ c ′And the opacity α is multiplied together to obtain the other single-point Gaussian distributions passed by the ray projected from the i-th single-point Gaussian distribution to the pixel point in the rasterized image, which is recorded as j.

[0114] 3. Densified rendering: By designing a preset loss function between the rasterized image and the real image (taken by the camera), and minimizing the function value of the preset loss function, the parameters such as the ellipsoid shape Σ, color (spherical harmonic function SH), and opacity α can be optimized. Among them, the preset loss function is defined as Among them, RGB m Represents the RGB value of the mth point in the rasterized image, RGB gt Represents the RGB value of the mth point in the input image (key frame). By minimizing L ct (Multi-channel camera L ct The point cloud is then densified and rendered into a 3D Gaussian distribution.

[0115] Parking space frame detection and occupied space detection: After dense rendering, 3D parking space frame detection is performed on the real-time reconstructed parking scene. Current 3D object detection methods are all suitable for point cloud detection and voxelization methods, which are difficult to apply to the 3D Gaussian module. Therefore, a pre-processing module is added to voxelnet (a pre-trained 3D point cloud parking space detection model). This pre-processing module can voxelize the input 3D Gaussian distribution and feed it into voxelnet for 3D parking space frame detection. For example, the mean μ in the 3D Gaussian distribution can be used to construct an X×Y×Z voxel grid, where the resolution of the voxel grid is a preset size, such as 0.02m. Therefore, for each voxel in X×Y×Z, the probability density and opacity of the single-point Gaussian distribution corresponding to the voxel are used to determine the occupancy value. The specific formula has been explained in the above example and will not be repeated here. Therefore, through the above pre-processing modules, the three-dimensional Gaussian distribution can be voxelized. Finally, 3D parking space detection can be performed through VoxelNet, and then a 3D parking space detection frame (parking space detection frame) can be obtained, which can be applied to other downstream tasks in the parking process.

[0116] The parking space recognition method provided in the embodiment of the present application converts each point cloud point in each sparse point cloud corresponding to each camera into a single-point Gaussian distribution according to initial parameters, determines the initial Gaussian distribution according to each single-point Gaussian distribution, determines the rasterized image corresponding to each camera according to the initial Gaussian distribution and the camera pose of each camera, and projects the initial Gaussian distribution onto the camera screen, updates each single-point Gaussian distribution based on the rasterized image corresponding to each camera and the key frame corresponding to the rasterized image based on a preset loss function, improves the matching degree between the single-point Gaussian distribution and the actual scene, and determines a three-dimensional Gaussian distribution based on the updated single-point Gaussian distribution, performs voxel processing on the three-dimensional Gaussian distribution according to a preset resolution, and obtains each voxel to provide suitable input for the three-dimensional point cloud parking space detection model, thereby realizing the construction of a three-dimensional Gaussian distribution matching the actual scene, improving the accuracy of point cloud reconstruction, and improving the accuracy of subsequent parking space detection.

[0117] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 400 includes one or more processors 401 and a memory 402 .

[0118] The processor 401 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0119] The memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may run the program instructions to implement the parking space recognition method of any embodiment of the present application described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0120] In one example, electronic device 400 may further include an input device 403 and an output device 404, which are interconnected via a bus system and / or other connection mechanisms (not shown). Input device 403 may include, for example, a keyboard, a mouse, etc. Output device 404 may output various information to the outside, including warning information, braking force, etc. Output device 404 may include, for example, a display, a speaker, a printer, a communication network, and remote output devices connected thereto.

[0121] Of course, to simplify, Figure 4 Only some of the components related to the present application in the electronic device 400 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, the electronic device 400 may further include any other appropriate components according to specific application scenarios.

[0122] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the parking space identification method provided by any embodiment of the present application.

[0123] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0124] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps of the parking space identification method provided by any embodiment of the present application.

[0125] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0126] It should be noted that the terms used in this application are only for describing specific embodiments and are not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates an exception, the words "one", "an", "a kind of" and / or "the" do not specifically refer to the singular and may also include the plural. The terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "comprise a..." do not exclude the presence of other identical elements in the process, method or device comprising the elements.

[0127] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0128] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. The above is only the preferred implementation method of this application. It should be pointed out that due to the limitations of textual expression, there are objectively infinite specific structures. For ordinary technicians in this technical field, without departing from the principles of the present invention, they can also make several improvements, modifications or changes, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the inventive concept and technical solution to other occasions without improvement, should be regarded as the scope of protection of this application.

Claims

1. A parking space recognition method, characterized in that: include: For each camera, obtain each key frame group in the video frame captured by the camera, and for each key frame group, determine each first feature point corresponding to the key frame group and each second feature point corresponding to the first feature point; wherein the key frame group is a combination of every two adjacent key frames in the video frame; Determining a camera pose of the camera according to each first feature point and a second feature point corresponding to each first feature point; Determining, according to the camera pose of the camera and the first feature points, a sparse point cloud corresponding to each first feature point; Determine a three-dimensional Gaussian distribution based on each point cloud point in each sparse point cloud corresponding to each camera, and voxelize the three-dimensional Gaussian distribution to determine each voxel and its corresponding occupancy value; The parking space detection frame corresponding to the key frame group is determined according to the occupancy value of each voxel and a pre-trained three-dimensional point cloud parking space detection model.

2. The method according to claim 1, characterized in that The determining the camera pose of the camera according to each first feature point and the second feature point corresponding to each first feature point includes: Determining an initial relative pose based on the first feature points and the second feature points corresponding to the first feature points using a structure-from-motion algorithm; Obtaining motor speed and inertial measurement information, and determining a Kalman relative posture based on the motor speed and the inertial measurement information; The camera pose of the camera is determined according to the initial relative pose and the Kalman relative pose.

3. The method according to claim 2, characterized in that After determining the camera pose of the camera according to the initial relative pose and the Kalman relative pose, the method further includes: Based on the feature points in the key frames of each camera at consecutive preset moments and the three-dimensional points corresponding to each feature point, the preset objective function is minimized and the camera pose of each camera is updated; wherein the preset objective function is constructed based on the error between the conversion points corresponding to each three-dimensional point commonly observed by each camera and the feature points in the key frames of each camera at each preset moment.

4. The method according to claim 3, characterized in that The preset objective function is: Among them, P i is the i-th 3D point among all 3D points observed by all cameras, N is the total number of 3D points, M is the total number at the preset time, and C is the total number of cameras. is the effective value of the i-th 3D point observed by the c-th camera at the t-th moment, m itc is the feature point corresponding to the i-th 3D point of the c-th camera at the t-th moment, ||·|| huber is the smooth L1 norm, is the preset external parameter transformation matrix of the c-th camera, is the homogeneous transformation matrix to be solved, π c is the local camera model of the c-th camera, is the transformation point corresponding to the i-th 3D point, F 1c F 2c is the camera pose of the c-th camera.

5. The method according to claim 1, wherein Determining a three-dimensional Gaussian distribution based on each point cloud point in each sparse point cloud corresponding to each camera includes: According to the initial parameters, each point cloud point in each sparse point cloud corresponding to each camera is converted into a single-point Gaussian distribution, and the initial Gaussian distribution is determined based on each single-point Gaussian distribution; Determining a rasterized image corresponding to each camera according to the initial Gaussian distribution and the camera pose of each camera; According to the rasterized images corresponding to the cameras and the key frames corresponding to the rasterized images, each single-point Gaussian distribution is updated based on a preset loss function, and a three-dimensional Gaussian distribution is determined based on the updated single-point Gaussian distribution.

6. The method according to claim 5, characterized in that The voxelizing the three-dimensional Gaussian distribution to determine each voxel and an occupancy value corresponding to each voxel includes: voxelize the three-dimensional Gaussian distribution according to a preset resolution to obtain voxels; For each voxel, the occupancy value of the voxel is determined according to the updated single-point Gaussian distributions corresponding to the voxel.

7. The method according to claim 6, characterized in that The determining the occupancy value of the voxel according to the updated single-point Gaussian distribution corresponding to the voxel includes: The occupancy value of the voxel is determined based on the following formula: Among them, v j is the occupancy value of the j-th voxel, N is the number of updated single-point Gaussian distributions corresponding to the voxel, α i is the opacity of the updated i-th single-point Gaussian distribution corresponding to the voxel, β i is the probability density of the updated i-th single-point Gaussian distribution corresponding to the voxel.

8. The method according to claim 1, characterized in that Determining the parking space detection frame corresponding to the key frame group according to the occupancy value of each voxel and a pre-trained three-dimensional point cloud parking space detection model includes: For each voxel, in response to the occupancy value of the voxel being greater than a preset value, the occupancy value of the voxel is updated to 1; in response to the occupancy value of the voxel being less than or equal to the preset value, the occupancy value of the voxel is updated to 0; The updated occupancy value of each voxel is input into a pre-trained three-dimensional point cloud parking space detection model to obtain a parking space detection frame corresponding to the key frame group.

9. An electronic device, characterized in that: The electronic device comprises: processor and memory; The processor is configured to execute the steps of the parking space recognition method according to any one of claims 1 to 8 by calling the program or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the steps of the parking space recognition method according to any one of claims 1 to 8.

Images