Point cloud extraction method and device based on parking scene, and parking method and device

Abstract

The invention provides a point cloud extraction method and device based on a parking scene, and a parking method and device, and the method comprises the steps: determining a transformation matrix of a camera pose of a vehicle according to the wheel speed pulse data and IMU data of the vehicle; based on an edge detection model, edge detection is carried out on a pre-collected parking image to obtain an edge detection result, and the edge detection result comprises obstacle detection points about static obstacles; and acquiring a multi-frame image tracking result of the obstacle detection point according to the transformation matrix of the camera pose, and acquiring depth information of the obstacle detection point according to the multi-frame image tracking result so as to acquire point cloud data corresponding to the parking image based on the depth information. Through the method, the point cloud extraction precision in the parking scene can be effectively improved, and the calculation amount in the point cloud extraction process is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a point cloud extraction method based on a parking scene, a parking method and device. BACKGROUND

[0002] With the development of automatic driving technology, the advanced driver assistance system (ADAS) gradually becomes the focus.

[0003] Among them, automatic parking (APA) as an important part of the ADAS system, its main function is to automatically perceive the surrounding environment, help the driver to automatically park and avoid collision risk. The current automatic parking system mainly relies on ultrasonic sensors or radar sensors to detect the point cloud data of the surrounding obstacles, which has certain limitations in precision and resolution, especially in complex urban environment, dim environment.

[0004] Therefore, it is urgent to propose a technical solution that can effectively improve the point cloud extraction precision in the parking scene to solve the above technical problems. SUMMARY

[0005] The present application provides a point cloud extraction method based on a parking scene, a parking method and device, which can effectively improve the point cloud extraction precision in the parking scene and reduce the calculation amount in the point cloud extraction process.

[0006] According to an aspect of the present application, a point cloud extraction method based on a parking scene is provided, comprising:

[0007] determining a transformation matrix of a camera pose of a vehicle according to wheel speed pulse data and IMU data of the vehicle;

[0008] performing edge detection on a pre-acquired parking image based on an edge detection model to obtain an edge detection result; wherein the parking image is an image collected for a static obstacle when the vehicle is parking, and the edge detection result includes an obstacle detection point of the static obstacle

[0009] obtaining a multi-frame image tracking result of the obstacle detection point according to the transformation matrix of the camera pose, and obtaining depth information of the obstacle detection point according to the multi-frame image tracking result, to obtain point cloud data corresponding to the parking image based on the depth information.

[0010] In an embodiment, the transformation matrix of the camera pose is determined according to the wheel speed pulse data and the IMU data of the vehicle, comprising:

[0011] According to wheel speed pulse data of a wheel, a linear velocity change of the vehicle is obtained;

[0012] According to IMU data, an acceleration change and / or an angular velocity change of the vehicle are obtained;

[0013] Fusion of the linear velocity change, the acceleration and / or the angular velocity change, a transformation matrix of the camera pose is determined.

[0014] In an embodiment, the fusion of the linear velocity change, the acceleration and / or the angular velocity change, the transformation matrix of the camera pose includes:

[0015] Through a Kalman filtering algorithm, the linear velocity change, the acceleration change and / or the angular velocity change are fused to obtain a fused linear velocity change, a fused acceleration change and / or a fused angular velocity change;

[0016] The fused linear velocity change, the fused acceleration change and / or the fused angular velocity change are input into a vehicle kinematics model to generate an incremental transformation matrix describing a change in vehicle pose;

[0017] According to the incremental transformation matrix, the transformation matrix of the camera pose is obtained.

[0018] In an embodiment, the obtaining of the multi-frame image tracking result of the obstacle detection point according to the transformation matrix of the camera pose includes:

[0019] Obtaining a first position of the obstacle detection point in the parking image at a first time;

[0020] According to the transformation matrix of the camera pose, a second position of the first position in the parking image at a second time is determined;

[0021] According to the matching detection point corresponding to the second position, the multi-frame image tracking result of the obstacle detection point is obtained.

[0022] In an embodiment, the determination of the second position of the first position in the parking image at the second time according to the transformation matrix of the camera pose includes:

[0023] According to the transformation matrix of the camera pose, an epipolar line of the first position in the parking image at the second time is calculated; wherein the epipolar line is a line formed by a set of similar positions of the first position in the parking image at the second time;

[0024] According to the epipolar line search algorithm, the matching detection point of the obstacle detection point is searched according to the epipolar line to determine the second position.

[0025] In an embodiment, the calculating, according to the transformation matrix of the camera pose, of the epipolar line of the first position in the second time in the parking image comprises:

[0026] determining, based on the transformation matrix of the camera pose and the camera intrinsic matrix, a fundamental matrix about the plurality of frames of images, the fundamental matrix being used to describe the geometric relationship between the parking image in the first time and the parking image in the first time;

[0027] calculating, according to the fundamental matrix and the homogeneous coordinates of the first position, the epipolar line of the first position in the second time in the parking image.

[0028] In an embodiment, the obtaining, according to the depth information, of the point cloud data corresponding to the parking image comprises:

[0029] converting, according to the depth information and the pixel coordinates of the obstacle detection point, to the three-dimensional coordinates in the camera coordinates by the camera intrinsic matrix;

[0030] converting, by the camera extrinsic matrix, the three-dimensional coordinates in the camera coordinates to the three-dimensional coordinates in the world coordinate system to obtain the point cloud data corresponding to the parking image.

[0031] In an embodiment, after the obtaining, based on the depth information, of the point cloud data corresponding to the parking image, further comprising:

[0032] dividing a plurality of grid cells; wherein each grid cell has a predetermined size;

[0033] assigning each point in the point cloud data to the corresponding grid cell according to the three-dimensional coordinates of each point;

[0034] for each grid cell, obtaining point cloud position information according to the weighted average position of all points in the grid cell;

[0035] obtaining processed point cloud data according to the point cloud position information of each grid cell.

[0036] According to a second aspect of the present application, a parking method is provided, comprising:

[0037] in response to a parking trigger signal of a vehicle, obtaining point cloud data corresponding to a pre-acquired parking image, the point cloud data being obtained according to a transformation matrix of a camera pose of the vehicle, a plurality of frames of images tracking results of obstacle detection points detected by an edge detection model, depth information of the obstacle detection points obtained according to the plurality of frames of images tracking results, and processed according to the depth information;

[0038] According to the point cloud data, a static obstacle in the vehicle parking scene is identified;

[0039] According to the identification result of the static obstacle, a parking system of the vehicle is controlled to park. According to a third aspect of the present application, a point cloud extraction device based on a parking scene is provided, comprising:

[0040] A matrix determination module is configured to determine a transformation matrix of a camera pose according to wheel speed pulse data and IMU data of a vehicle;

[0041] An edge detection module is configured to perform edge detection on a pre-acquired parking image based on an edge detection model to obtain an edge detection result, wherein the parking image is an image acquired for a static obstacle when the vehicle is parking, and the edge detection result includes an obstacle detection point about the static obstacle

[0042] A depth acquisition module is configured to acquire a multi-frame image tracking result of the obstacle detection point according to the transformation matrix of the camera pose, and acquire depth information of the obstacle detection point according to the multi-frame image tracking result;

[0043] A point cloud extraction module is configured to acquire point cloud data corresponding to the parking image based on the depth information.

[0044] In an embodiment, the matrix determination module comprises:

[0045] A first acquisition unit is configured to acquire a linear velocity change of the vehicle according to wheel speed pulse data of a wheel;

[0046] A second acquisition unit is configured to acquire an acceleration change and / or an angular velocity change of the vehicle according to IMU data;

[0047] A fusion unit is configured to fuse the linear velocity change, the acceleration change and / or the angular velocity change to determine a transformation matrix of a camera pose.

[0048] In an embodiment, the fusion unit is specifically configured to:

[0049] The linear velocity change, the acceleration change and / or the angular velocity change are fused by a Kalman filtering algorithm to obtain a fused linear velocity change, a fused acceleration change and / or a fused angular velocity change;

[0050] The fused linear velocity change, the fused acceleration change and / or the fused angular velocity change are input into a vehicle kinematics model to generate an incremental transformation matrix describing a change in vehicle pose;

[0051] The transformation matrix of the camera pose is obtained according to the incremental transformation matrix.

[0052] In an implementation, the depth acquisition module comprises:

[0053] a third acquisition unit, configured to acquire a first position of the obstacle detection point in the parking image at the first time;

[0054] a position determination unit, configured to determine a second position of the first position in the parking image at the second time according to a transformation matrix of the camera pose;

[0055] a fourth acquisition unit, configured to acquire a multi-frame image tracking result of the obstacle detection point according to a matching detection point corresponding to the second position.

[0056] In an implementation, the position determination unit is specifically configured to:

[0057] calculate an epipolar line of the first position in the parking image at the second time according to a transformation matrix of the camera pose; wherein the epipolar line is a line formed by a set of similar positions of the first position in the parking image at the second time;

[0058] determine the second position according to a matching detection point of the obstacle detection point searched according to the epipolar line.

[0059] In an implementation, the calculation of the epipolar line of the first position in the parking image at the second time according to the transformation matrix of the camera pose is specifically:

[0060] determine a fundamental matrix about the multi-frame images based on the transformation matrix of the camera pose and a camera intrinsic matrix, the fundamental matrix being used to describe a geometric relationship between the parking image at the first time and the parking image at the first time;

[0061] calculate the epipolar line of the first position in the parking image at the second time according to the fundamental matrix and a homogeneous coordinate of the first position.

[0062] In an implementation, the apparatus further comprises:

[0063] a fifth acquisition unit, configured to acquire a camera extrinsic parameter according to the transformation matrix of the camera pose, the camera extrinsic parameter comprising a rotation matrix and a translation vector;

[0064] The point cloud extraction module is specifically configured to:

[0065] convert the depth information and the pixel coordinate of the obstacle detection point to a three-dimensional coordinate in the camera coordinate by a camera intrinsic matrix;

[0066] The three-dimensional coordinates in the camera coordinates are converted to three-dimensional coordinates in a world coordinate system through a camera extrinsic matrix to obtain point cloud data corresponding to the parking image.

[0067] In an embodiment, the apparatus further comprises:

[0068] a grid dividing module configured to divide a plurality of grid cells, wherein each grid cell has a predetermined size;

[0069] a point cloud assigning module configured to assign each point in the point cloud data to a corresponding grid cell according to the three-dimensional coordinates of the point;

[0070] a position obtaining module configured to obtain point cloud position information for each grid cell according to a weighted average position of all points in the grid cell;

[0071] a point cloud processing module configured to obtain processed point cloud data according to the point cloud position information of each grid cell.

[0072] According to a fourth aspect of the present application, a parking apparatus is provided, comprising:

[0073] a point cloud obtaining module configured to obtain, in response to a parking trigger signal of a vehicle, pre-acquired point cloud data corresponding to parking images, the point cloud data being obtained according to a transformation matrix of a camera pose of the vehicle, after a plurality of image tracking results of obstacle detection points detected by an edge detection model are obtained, depth information of the obstacle detection points is obtained according to the plurality of image tracking results, and the point cloud data is determined based on the depth information;

[0074] a recognition module configured to recognize static obstacles in a parking scene of the vehicle according to the point cloud data;

[0075] a parking module configured to control a parking system of the vehicle to park according to a recognition result of the static obstacles.

[0076] According to a fifth aspect of the present application, an electronic device is provided, comprising a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the electronic device executes the method provided in any one of the above first aspects.

[0077] According to a fourth aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium stores computer execution instructions, the computer execution instructions are executed by a processor to implement the method provided in any one of the above first aspects.

[0078] According to a fifth aspect of this application, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the method provided in any of the first aspects above.

[0079] The point cloud extraction method, parking method, and apparatus provided in this application replace the point cloud acquisition method based on SLAM technology in related technologies. It directly utilizes the vehicle's wheel speed pulse data and IMU data to calculate the camera pose transformation matrix, eliminating the need for matrix calculations based on image data, significantly reducing computational load, and avoiding scale scaling issues while achieving high accuracy. Simultaneously, it combines an edge detection model to perform edge detection on the image and uses multi-frame image tracking to calculate the depth information of obstacle detection points (for static obstacles, especially static suspended obstacles). This improves the recognition accuracy of obstacle detection points in environments with limited texture information or poor lighting, effectively enhancing the detection accuracy of obstacle point clouds and reducing the collision risk in dimly lit or low-contrast parking scenarios. Attached Figure Description

[0080] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0081] Figure 1 This is a schematic diagram of a possible scenario provided for an embodiment of this application;

[0082] Figure 2 A flowchart illustrating a point cloud extraction method based on a parking scenario provided in this application embodiment;

[0083] Figure 3 for Figure 2 A flowchart illustrating step S201;

[0084] Figure 4 A flowchart illustrating another point cloud extraction method based on a parking scenario provided in this application embodiment;

[0085] Figure 5a Example image of a dimly lit parking scene;

[0086] Figure 5b Example image of point cloud extracted using embodiments of this application in a dimly lit parking scene;

[0087] Figure 6 A schematic flowchart illustrating a parking method provided in an embodiment of this application;

[0088] Figure 7 A schematic diagram of a point cloud extraction device based on a parking scenario provided in an embodiment of this application;

[0089] Figure 8 A structural schematic diagram of a parking device provided by an embodiment of the present application is shown in the following figure.

[0090] Figure 9 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in the following figure.

[0091] Figure 10 A structural diagram of a server provided by an example embodiment of the present application is shown in the following figure.

[0092] The specific embodiments of the present application have been shown in the above figures, and will be described in more detail hereinafter. These figures and the written description are not intended to restrict the scope of the present application in any way, but to illustrate the present application by reference to specific embodiments. DETAILED DESCRIPTION

[0093] The example embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0094] The embodiments of the present application will be explained in the following with reference to application scenarios. The point cloud extraction method based on a parking scenario provided by the embodiments of the present application can be applied to the application scenario of intelligent driving, and more specifically, can be applied to the automatic driving application scenario based on vehicle cloud computing. For example, the execution subject of the method provided by the embodiments of the present application can be a server or a vehicle. The following will take the server as the execution subject of the method provided by the embodiments of the present application.

[0095] Figure 1 A scenario schematic diagram of a point cloud extraction method based on a parking scenario provided by an embodiment of the present application is shown in the following figure. Figure 1As shown, the server 110 and the intelligent vehicle 120 are connected via a network, the server 110 is built-in with an edge detection model, and can obtain vehicle data (such as wheel speed pulse data, IMU data, and image data, etc.) of the intelligent vehicle 120, extract obstacle point cloud based on the related data of the intelligent vehicle 120, and transmit the obstacle point cloud to the intelligent vehicle 120, so that the intelligent vehicle 120 can use the point cloud data to realize automatic parking, such as constructing a three-dimensional model of obstacles around the intelligent vehicle 120 based on the point cloud data, so as to realize automatic parking by avoiding obstacles. Alternatively, the server 110 can also construct a three-dimensional model of obstacles around the intelligent vehicle 120 based on the extracted point cloud data, and further identify a parking space, and send the identification result to the intelligent vehicle 120, so that the intelligent vehicle 120 further realizes automatic parking. The server 110 can be a physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, and other basic cloud computing services.

[0096] In related technologies, the point cloud data of the surrounding obstacles is mainly detected by ultrasonic sensors or radar sensors. The ultrasonic sensors or radar sensors can usually only detect two-dimensional plane or low-resolution point cloud data, and it is difficult to detect static obstacles with height, especially suspended static obstacles, which is prone to problems of low detection accuracy and resolution. The point cloud information is extracted by using a method based on visual positioning and mapping (Advanced Driver Assistance Systems, SLAM) and three-dimensional scene reconstruction technology. The SLAM technology extracts and matches feature points through image frames captured by a camera, calculates a transformation matrix of the camera pose, and estimates the depth of the feature points based on the matrix, thereby generating point cloud information of the environment. However, although the SLAM technology provides a new idea for environmental perception, it still has various drawbacks in practical application. On the one hand, the SLAM technology relies on image data to calculate the camera pose transformation matrix and generate point cloud information, and this process involves a large amount of calculation, which may cause the system to fail to meet the strict real-time requirements of automatic driving. On the other hand, due to the lack of scale information in the SLAM technology (in the SLAM system, especially in the visual SLAM based on a monocular camera, the depth perception ability is lacking, so it is difficult to obtain the real scale of objects in the scene), for static obstacles with height, especially suspended static obstacles, the generated point cloud data has limited accuracy, which is difficult to meet the high accuracy requirements in the automatic parking scene. In addition, the positioning method based on vision may have degeneration problems in some scenes, for example, in an environment with less texture information or poor lighting conditions, the reliability and accuracy of visual positioning may be significantly reduced.

[0097] Therefore, the method for extracting point cloud based on a parking scene, the parking method and the device provided by the embodiments of the present application determine the transformation matrix of the camera pose of the vehicle according to the wheel speed pulse data and the IMU data of the vehicle, and perform edge detection on the pre-acquired parking image based on an edge detection model to obtain an edge detection result. The parking image is an image acquired for a static obstacle when the vehicle is parked, and the edge detection result includes an obstacle detection point about the static obstacle. Then, the multi-frame image tracking result of the obstacle detection point is obtained according to the transformation matrix of the camera pose, and the depth information of the obstacle detection point is obtained according to the multi-frame image tracking result, so as to obtain the point cloud data corresponding to the parking image based on the depth information. In this process, instead of the point cloud acquisition mode based on the SLAM technology in the related art, the transformation matrix of the camera pose is directly calculated by using the wheel speed pulse data and the IMU data of the vehicle, without relying on image data for matrix calculation, so that the calculation amount is greatly reduced, the scale scaling problem does not exist, and the precision index is higher. At the same time, the image is edge-detected by combining the edge detection model, and the depth information of the obstacle detection point is calculated through multi-frame image tracking, so that the recognition accuracy of the obstacle detection point can be improved in an environment with less texture information or poor light, the detection accuracy of the obstacle point cloud is effectively improved, and the collision risk in a dim or low-contrast parking scene is reduced.

[0098] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below in combination with the drawings and specific embodiments. It should be noted that these specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments.

[0099] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0100] Figure 2 The flowchart of the method for extracting point cloud based on a parking scene provided by the embodiments of the present application can be applied to the server 120 in Figure 1 , as shown in Figure 2 , the method comprises steps S201-S203.

[0101] Step S201, determining a transformation matrix of a camera pose of a vehicle according to wheel speed pulse data and IMU data of the vehicle.

[0102] Exemplarily, wheel speed pulse data can be collected by wheel speed sensors, and inertial measurement unit (IMU) data, i.e., acceleration and angular velocity information of the vehicle, can be collected by an inertial measurement unit (accelerometer and gyroscope).

[0103] It is found through research that, in actual applications, the camera pose is approximately the same as the vehicle pose. Based on this finding, the present embodiment replaces the SLAM technology to calculate the camera pose transformation matrix and generate point cloud information through image data, combines the wheel speed pulse data and IMU data of the vehicle, and calculates the pose change of the ego motion, i.e., the pose change of the camera, so as to effectively reduce the calculation amount of the camera pose transformation matrix. It can be understood that the transformation matrix of the camera pose, i.e., the transformation matrix of the camera pose at the corresponding time of the two frames of images.

[0104] In an embodiment, as shown in Figure 3 The step S201 of determining the transformation matrix of the camera pose according to the wheel speed pulse data and the IMU data of the vehicle can adopt the following manner:

[0105] The step S2011 obtains the linear velocity change of the vehicle according to the wheel speed pulse data of the wheels.

[0106] Exemplarily, the wheel speed pulse data is obtained from the wheel speed sensors of the wheels, and the linear velocity change of the vehicle can be obtained by integrating the wheel speed pulse data. Specifically, each pulse can correspond to a fixed angle rotation of the wheel, and the linear velocity of the wheel can be calculated by combining the known radius of the wheel and the pulse count. The linear velocity change of the vehicle can be calculated by integrating the linear velocity and comparing the linear velocities at different time points.

[0107] In some examples, the vehicle speed signal can directly provide the linear velocity information of the vehicle, and the linear velocity change of the vehicle can also be directly obtained based on the vehicle speed information.

[0108] The step S2012 obtains the acceleration change and / or angular velocity change of the vehicle according to the IMU data.

[0109] The IMU provides acceleration and angular velocity information, and the embodiment can integrate these data to obtain the velocity change and angle change of the vehicle.

[0110] The step S2013 fuses the linear velocity change, the acceleration change and / or the angular velocity change to determine the transformation matrix of the camera pose.

[0111] Exemplarily, Kalman filter can be used to fuse these data to correct errors and improve the accuracy of pose estimation, and a 4x4 homogeneous transformation matrix is formed to describe the rotation and translation of the camera. For example, the camera position Poscur and Pos pre between the transformation matrix where R represents a rotation matrix, and t represents a translation vector. Based on the timestamps of the images and the timestamps of the sensor signals, the transformation matrix T1 of the camera pose at the corresponding time of the two images can be obtained by interpolation. The interpolation process can be referred to related technologies, and will not be described in detail here.

[0112] In some examples, the transformation matrix of the camera pose can be determined by fusing the linear velocity change, the acceleration change, and / or the angular velocity change in the above steps in the following manner: the linear velocity change, the acceleration change, and / or the angular velocity change are fused by a Kalman filtering algorithm to obtain fused linear velocity change, acceleration change, and / or angular velocity change; the fused linear velocity change, acceleration change, and / or angular velocity change are input into a vehicle kinematics model to generate an incremental transformation matrix describing the change of the vehicle pose; and the transformation matrix of the camera pose is obtained according to the incremental transformation matrix.

[0113] For example, in the Kalman filtering process, a state vector can be defined to include the change quantities of the linear velocity change, the acceleration change, and the angular velocity change. Then, a state transition model and an observation model are established to respectively describe the evolution of the change quantities and the relationship between the sensor measurements. In the filtering process, the state transition model is used in the prediction step to estimate the change quantities and error covariance at the next time step, and the sensor observation data can be used in the update step to correct the prediction result. By calculating the Kalman gain, the filter finds a better balance between prediction and observation, and provides a better estimate of the fused change quantities. This method significantly improves the accuracy of the motion state change estimation, especially in the presence of sensor data noise.

[0114] For example, the vehicle kinematics model can use a bicycle model or a more complex dynamics model. By inputting the fused linear velocity change, acceleration change, angular velocity change, and other motion change quantities into the dynamics model, the kinematics model can calculate the translation (displacement along the x, y, z axes) and rotation (rotation around the x, y, z axes) of the vehicle within a time step according to the input change quantities. By combining the calculated translation and rotation information into a 4x4 homogeneous transformation matrix, i.e., an incremental transformation matrix, the incremental transformation can be used to describe the change of the vehicle pose within a time step. The incremental transformation matrix can be converted into the transformation matrix of the camera pose. For example, the change of the camera pose is converted from the change of the vehicle pose by using a fixed transformation matrix of the camera relative to the vehicle.

[0115] In some embodiments, in addition to the above-mentioned manner of determining the transformation matrix of the camera pose, an inertial navigation system (INS) calibration can also be adopted, using IMU data for inertial navigation, estimating the change in speed and position of the vehicle by double integrating the acceleration and angular velocity, and then correcting the change in speed in combination with the wheel speed pulse data to determine the transformation matrix of the camera pose. Alternatively, an adaptive filter (such as an adaptive Kalman filter) can also be used to dynamically adjust the weight of the sensor data, and adjust the fusion strategy of the wheel speed and IMU data according to the real-time environment and motion state, to improve the robustness of the pose estimation.

[0116] In some embodiments, more source data such as wheel speed pulses, vehicle speed, steering wheel angle, IMU, etc. can also be combined to calculate the transformation matrix of the camera pose. For example, by filtering the noise of these sensor data, and using the vehicle kinematic model in combination with the Kalman filter, the wheel speed, steering angle and IMU information are fused to estimate the state of the vehicle, to further improve the accuracy of the transformation matrix of the camera pose.

[0117] Step S202, performing edge detection on the pre-acquired parking image based on an edge detection model to obtain an edge detection result; wherein the parking image is an image acquired for a static obstacle when the vehicle is parked, and the edge detection result includes an obstacle detection point about the static obstacle.

[0118] Illustratively, the parking image can be acquired by one or more fisheye cameras of the vehicle. The fisheye camera has a wide viewing angle, which can reach 180 degrees or even wider. Such a wide-angle view allows the camera to capture a larger range of environmental information at a single location, making it suitable for monitoring the situation around the vehicle, especially in narrow or complex parking environments. The fisheye camera can acquire a panoramic view of the vehicle in the parking scene, effectively avoiding missed identification of obstacles.

[0119] Further, after image acquisition by multiple fisheye cameras, the parking image can be rectified. Specifically, the fisheye image is first preprocessed (such as denoising, color correction, etc.), and based on the surround view calibration data, which can include a camera intrinsic matrix K, the fisheye distortion and image eccentricity can be corrected. The rectified image is used as the parking image, which is input into the edge detection model to realize the edge profile of each static obstacle in the parking image (i.e. the edge profile plane formed by the obstacle detection points).

[0120] In this embodiment, the edge detection model can use a deep learning model, such as a convolutional neural network (CNN). Optionally, to improve the accuracy of obstacle edge detection in parking scenarios, various images under parking scenarios (especially images containing complex urban environments or dim environments, such as low-light low-contrast images) can be used as sample data to train the deep learning model to train a model that meets the recognition accuracy of parking scenarios in various complex environments.

[0121] For example, a large number of parking scenario image data is collected, which should cover various complex urban environments and scenarios under different lighting conditions, such as low-light (such as underground garage, etc.) and low-contrast (such as foggy environment, etc.) environments. By annotating the collected images, the edges of the obstacles in the images are annotated. Use the convolutional neural network as the initial model to be trained, and use the annotated image data to train the model to train a model that meets the user's expectations (such as meeting the pre-defined loss function, and those skilled in the art can select a suitable loss function for edge detection in combination with actual application or prior data) or meets the number of iterations. After training is completed, the final performance of the model is evaluated using an independent test set to ensure that the model has good generalization ability on unseen data. The trained model is deployed in the server or the parking assistance system, and its performance is continuously monitored for necessary updates and maintenance. Through the above process, a parking scenario obstacle edge detection model with high accuracy in dim and low-contrast environments can be trained.

[0122] In step S203, the multi-frame image tracking results of the obstacle detection points are obtained according to the transformation matrix of the camera pose, and the depth information of the obstacle detection points is obtained according to the multi-frame image tracking results, so as to obtain the point cloud data corresponding to the parking image based on the depth information.

[0123] For example, after detecting the obstacle detection points in the above steps, these points can be tracked and matched between different frames. For example, a descriptor matching algorithm can be used to match the detection points between adjacent frames. The transformation matrix of the camera pose is used to calculate the epipolar constraint to reduce the matching search range and improve the matching accuracy. The depth information can be calculated by using a triangulation method to calculate the position of the obstacle detection points in the three-dimensional space to extract the depth information (i.e., the z-axis coordinate of the detection point).

[0124] In one embodiment, the step S203 of obtaining the multi-frame image tracking results of the obstacle detection points according to the transformation matrix of the camera pose can be performed in the following manner:

[0125] Obtain the first position of the obstacle detection point in the parking image at the first time;

[0126] determine a second position of the first position in the parking image at the second time according to the transformation matrix of the camera pose;

[0127] obtain the multi-frame image tracking result of the obstacle detection point according to the matching detection point corresponding to the second position.

[0128] Exemplarily, in the parking image at the first time (such as t1), the position of the obstacle detection point, that is, the first position, is recognized and obtained.

[0129] It has been mentioned above that the transformation matrix of the camera pose can describe the motion (including translation and rotation) information of the camera changing over time, and through the transformation matrix of the camera pose, the position of the first position in the image at the second time (such as t2), that is, the second position, can be determined. In the image at the second time, the detection point matched with the obstacle detection point is searched and determined with the second position as a reference, so as to efficiently obtain the multi-frame image tracking result of the obstacle detection point.

[0130] Specifically, in the above step of determining the second position of the first position in the parking image at the second time according to the transformation matrix of the camera pose, the following manner can be adopted:

[0131] calculating an epipolar line of the first position in the parking image at the second time according to the transformation matrix of the camera pose; wherein the epipolar line is a line formed by a set of similar positions of the first position in the parking image at the second time;

[0132] determining the second position by searching the epipolar line for the matching detection point of the obstacle detection point through an epipolar search algorithm.

[0133] It can be understood that the epipolar search algorithm is a method based on epipolar geometry for stereo matching. This method takes advantage of the camera geometric relationship to reduce the search range and improve the matching efficiency.

[0134] In this embodiment, the epipolar search algorithm is used to calculate the epipolar line of the first position in the parking image at the second time, and by searching on this epipolar line, the detection point matched with the obstacle detection point at the first time can be found. This process takes advantage of the camera geometric relationship to greatly reduce the search range and thus improve the matching efficiency.

[0135] Further, in the above step of calculating the epipolar line of the first position in the parking image at the second time according to the transformation matrix of the camera pose, the following manner can be adopted:

[0136] determining a fundamental matrix about the multi-frame images based on the transformation matrix of the camera pose and an intrinsic matrix of the camera, the fundamental matrix being used to describe the geometric relationship between the parking image at the first time and the parking image at the first time;

[0137] According to the basis matrix and the homogeneous coordinates of the first position, a polar line of the first position in the parking image at the second time is calculated.

[0138] Wherein, the camera intrinsic matrix is a known parameter, containing the focal length and optical center of the camera and other parameters. Through the rotation matrix and translation vector of the camera, the essential matrix of the camera can be calculated, and the basis matrix can be obtained by using the essential matrix and the camera intrinsic matrix. It can be understood that the homogeneous coordinates are a coordinate system for representing n-dimensional vectors with n+1-dimensional vectors, which is used in projective geometry to simplify the calculation of geometric problems.

[0139] Exemplarily, the obstacle detection points extracted by the edge detection model are matched between frames. That is, the obstacle detection points of the t1 time image and the t2 time image obtained by the edge detection model can be matched by the same obstacle detection points between the two frames of images through the polar line search method through the transformation matrix of the camera pose. After the t1 time obstacle detection point P1 passes through the camera conversion matrix, it may fall on L (L is the polar line of P1) of the t2 time image. By searching on L, the key point P2 that best matches P1 is found, that is, P1 and P2 are considered as the same point. Then, the depth estimation of P1 and P2 can be obtained through the principle of triangulation. For specific process, please refer to the related technology of depth estimation of triangular flower, which will not be repeated here.

[0140] Through the above technical solutions, the matching and tracking efficiency and accuracy of the obstacle detection points can be effectively improved.

[0141] Further, the point cloud data corresponding to the parking image can be obtained according to the depth information in the above step in the following way: according to the depth information and the pixel coordinates of the obstacle detection points, the three-dimensional coordinates in the camera coordinates are converted through the camera intrinsic matrix; the three-dimensional coordinates in the camera coordinates are converted to the three-dimensional coordinates in the world coordinate system through the camera extrinsic parameters, to obtain the point cloud data corresponding to the parking image.

[0142] After obtaining the depth information of each obstacle detection point, the pixel coordinates (i.e. x-axis and y-axis coordinates) of the obstacle detection points can be converted to the three-dimensional coordinates in the world coordinate system, i.e. 3D point cloud data, through the camera intrinsic and extrinsic parameters.

[0143] Wherein, the pixel coordinates of the obstacle detection points, i.e. the x and y axis coordinates, can be directly obtained from the pixel coordinates on the image plane through the projection model of the camera.

[0144] In some embodiments, the inter-frame matching of the obstacle detection point which is successfully matched is tracked for multiple frames. In the multi-frame tracking, a depth value can be obtained from the result of each inter-frame matching, and multiple depth estimation values are put into a depth filter to estimate the depth, so that a reliable depth value can be obtained (e.g., by calculating the mean value). Alternatively, for the obstacle detection point which is successfully matched in the continuous multi-frame matching, it is considered that the existence probability is high and the depth estimation is accurate. Through the internal and external parameters of the camera, the depth value of the obstacle point can be converted into the three-dimensional coordinate value of the point cloud in the world coordinate system. In this way, the extraction accuracy and efficiency of the point cloud data can be effectively improved.

[0145] Figure 4 is a flowchart of another point cloud extraction method based on a parking scene provided by an embodiment of the present application. On the basis of the above-mentioned embodiment, the present embodiment performs post-processing on the extracted point cloud data to reduce the redundancy and noise of the point cloud data and further improve the accuracy of the point cloud data. Specifically, in addition to the steps S201-S203, the present embodiment can further include the following steps S401-S404 in step S203.

[0146] Step S401, divide a plurality of grid cells; wherein each grid cell has a predetermined size.

[0147] Considering that in the above-mentioned point cloud extraction process, there can be multiple frames of observation results for the edge of the same object, i.e., including multiple redundant detection points, the present embodiment performs post-processing on the point cloud data. The present embodiment adopts a point cloud data gridding processing method to reduce the redundancy of the point cloud data.

[0148] Optionally, the range of the grid can be determined according to the three-dimensional coordinates of the point cloud data in the world coordinate system, so as to divide a plurality of grid cells. For example, the bounding box of the point cloud data can be determined by calculating the minimum and maximum coordinates of all points in the point cloud, so as to define the range of the grid, and then divide a plurality of grid cells in the range of the grid.

[0149] It should be noted that the size of the grid cell can be set by the person skilled in the art according to the actual application or experience value, for example, the size of the grid is 10cm*10cm.

[0150] Step S402, according to the three-dimensional coordinates of each point in the point cloud data, each point is assigned to the corresponding grid cell.

[0151] Exemplarily, the three-dimensional coordinates (x, y, z) of each point in a set of point cloud data, for each point, the grid cell index to which the point belongs is calculated according to the coordinates of the point, that is, the coordinates of the point are converted into the row and column numbers of the grid to determine which grid cell the point belongs to in the three-dimensional space, so as to assign the point to the corresponding grid cell.

[0152] In step S403, for each grid cell, the point cloud position information is obtained according to the weighted average position of all points in the grid cell.

[0153] Exemplarily, for each grid cell, the weighted average position of all points in the grid cell is calculated, wherein the weight of each point can be determined based on the attribute (such as reflection intensity) of the point or external information (such as deviation). By calculating the weighted average position of all points in each grid cell, the center point or representative point of the grid cell can be determined.

[0154] In step S404, the processed point cloud data is obtained according to the point cloud position information of each grid cell.

[0155] Specifically, a new point cloud data set can be created using the point cloud position information of each grid cell, that is, the weighted average position, which effectively reduces the redundancy of repeated point clouds.

[0156] Through the above scheme, for the same object edge, the algorithm can have multiple observation results, and the grid processing is performed on the multiple observation values. For example, by using a 10cm x 10cm grid, the discrete point cloud is processed by weighted average to obtain more accurate point cloud position information, and the noise points are filtered, further improving the point cloud accuracy.

[0157] To verify the effectiveness of the embodiments of the present application, as shown in Figure 5a and Figure 5b As shown in the actual automatic parking scene, the light in the garage is relatively dim, and there are obstacles such as suspended fire boxes beside the parking space to be parked. By using the technical scheme provided in the embodiments, the point cloud is extracted, as shown in Figure 5a , it can be clearly seen that there are obstacles such as suspended fire boxes beside the parking space to be parked in the parking scene. Figure 5b

[0158] ​Compared with the point cloud extraction scheme of the SLAM technology, the technical scheme provided by the embodiments of the present application does not need to rely on images to obtain a camera pose conversion matrix to extract point clouds, and a conversion matrix is fused and calculated based on a vehicle body wheel speed pulse signal and an IMU. Since the data source is a vehicle body sensor, compared with images, there is less data processing content, the amount of calculation is greatly reduced, and the real-time requirement in a parking scene can be met. In the point cloud extraction process, an edge detection algorithm is combined, obstacles in the environment can be stably extracted, low-light and low-contrast scenes in which the effect of a traditional algorithm is poor can be targeted for obstacle detection, so as to improve the point cloud extraction precision, and the model optimization can be applied to more different environments in the parking scene, and the software development amount is reduced.

[0159] Figure 6 is a flowchart of a parking method provided by the embodiments of the present application. The execution subject of the method can be a vehicle, as shown in the figure, the method comprises steps S601-S603. Figure 6

[0160] Step S601, in response to a parking trigger signal of the vehicle, point cloud data corresponding to a pre-acquired parking image is acquired, the point cloud data is obtained according to a conversion matrix of a camera pose of the vehicle, after a multi-frame image tracking result of an obstacle detection point detected by an edge detection model is acquired, depth information of the obstacle detection point is acquired according to the multi-frame image tracking result, and the conversion matrix is determined based on the depth information.

[0161] It should be noted that the specific principle and effect of the extraction process of the point cloud data can be referred to the embodiments above, and will not be repeated here.

[0162] Step S602, according to the point cloud data, a static obstacle in the parking scene of the vehicle is identified.

[0163] Exemplarily, the extracted point cloud data includes horizontal and vertical coordinate information and height information of the point cloud, and the vehicle can determine the position and shape of the static obstacle based on the point cloud data information. When there are multiple point cloud data, the point cloud data can be segmented into different clusters through clustering analysis, each cluster represents a potential obstacle, and the bounding box and other geometric features of each cluster are calculated to identify the shape and size of different obstacles.

[0164] ​In some examples, static obstacles in a vehicle parking scene can be identified by three-dimensional modeling of the point cloud data. For example, a three-dimensional model corresponding to the point cloud data (if classified into multiple clusters, a three-dimensional model for each cluster can be generated) is generated by a surface reconstruction technique such as a convex hull algorithm or triangulation, so that the generated three-dimensional model provides shape and structure information of the static obstacles. On this basis, geometric features such as volume, surface area and bounding box can be extracted to further analyze the shape and size of the obstacles. By calculating the centroid or bounding box of each three-dimensional model, the specific location of the obstacle in the environment can be determined.

[0165] Step S603, according to the identification result of the static obstacle, controlling the parking system of the vehicle to park.

[0166] After identifying the static obstacle, the parking system is controlled to park, so that the vehicle parks around the static obstacle during parking, improving the efficiency and safety of parking.

[0167] Figure 7 is a structural schematic diagram of a point cloud extraction device based on a parking scene provided by an embodiment of the present application, as shown in Figure 7 The device includes a matrix determination module 701, an edge detection module 702, a depth acquisition module 703, and a point cloud extraction module 701, wherein,

[0168] The matrix determination module 701 is configured to determine a transformation matrix of a camera pose based on wheel speed pulse data and IMU data of a vehicle.

[0169] The edge detection module 702 is configured to perform edge detection on a pre-acquired parking image based on an edge detection model to obtain an edge detection result; wherein the parking image is an image acquired for a static obstacle when the vehicle is parking, and the edge detection result includes obstacle detection points about the static obstacle. The depth acquisition module 703 is configured to acquire multi-frame image tracking results of the obstacle detection points based on the transformation matrix of the camera pose, and acquire depth information of the obstacle detection points based on the multi-frame image tracking results.

[0170] The point cloud extraction module 704 is configured to acquire point cloud data corresponding to the parking image based on the depth information.

[0171] In an embodiment, the matrix determination module 701 includes:

[0172] The first acquisition unit is configured to acquire a linear velocity change of the vehicle based on wheel speed pulse data of a wheel.

[0173] The second acquisition unit is configured to acquire an acceleration change and / or an angular velocity change of the vehicle based on IMU data.

[0174] a fusion unit configured to fuse the linear velocity variation, the acceleration variation and / or the angular velocity variation, and determine a transformation matrix of the camera pose.

[0175] In an embodiment, the fusion unit is specifically configured to:

[0176] fuse the linear velocity variation, the acceleration variation and / or the angular velocity variation by using a Kalman filtering algorithm, to obtain fused linear velocity variation, fused acceleration variation and / or fused angular velocity variation;

[0177] input the fused linear velocity variation, the fused acceleration variation and / or the fused angular velocity variation into a vehicle kinematics model, to generate an incremental transformation matrix describing a change in the vehicle pose;

[0178] obtain the transformation matrix of the camera pose according to the incremental transformation matrix.

[0179] In an embodiment, the depth acquisition module 703 comprises:

[0180] a third acquisition unit configured to acquire a first position of the obstacle detection point in the parking image at the first time;

[0181] a position determination unit configured to determine a second position of the first position in the parking image at the second time according to the transformation matrix of the camera pose;

[0182] a fourth acquisition unit configured to acquire a multi-frame image tracking result of the obstacle detection point according to a matching detection point corresponding to the second position.

[0183] In an embodiment, the position determination unit is specifically configured to:

[0184] calculate an epipolar line of the first position in the parking image at the second time according to the transformation matrix of the camera pose; wherein the epipolar line is a line formed by a set of similar positions of the first position in the parking image at the second time;

[0185] determine the second position by searching the epipolar line and the matching detection point of the obstacle detection point according to an epipolar line search algorithm.

[0186] In an embodiment, the calculation of the epipolar line of the first position in the parking image at the second time according to the transformation matrix of the camera pose is specifically:

[0187] determine a fundamental matrix about the multi-frame images based on the transformation matrix of the camera pose and an intrinsic matrix of the camera, the fundamental matrix being used to describe a geometric relationship between the parking image at the first time and the parking image at the first time;

[0188] According to the homogeneous coordinates of the base matrix and the first position, a polar line of the first position in a second time in the parking image is calculated.

[0189] In an implementation, the apparatus further includes:

[0190] The fifth acquisition unit is configured to acquire the camera extrinsic parameter according to the transformation matrix of the camera pose, the camera extrinsic parameter including a rotation matrix and a translation vector;

[0191] The point cloud extraction module 704 is specifically configured to:

[0192] According to the depth information and the pixel coordinates of the obstacle detection point, the three-dimensional coordinates in the camera coordinates are converted through the camera intrinsic matrix;

[0193] The three-dimensional coordinates in the camera coordinates are converted into the three-dimensional coordinates in the world coordinate system through the camera extrinsic parameter, so as to obtain the point cloud data corresponding to the parking image.

[0194] In an implementation, the apparatus further includes:

[0195] The grid division module is configured to divide a plurality of grid units; wherein each grid unit has a predetermined size;

[0196] The point cloud distribution module is configured to distribute each point in the point cloud data into a corresponding grid unit according to the three-dimensional coordinates of each point in the world coordinate system;

[0197] The position acquisition module is configured to acquire the point cloud position information according to the weighted average position of all points in each grid unit for each grid unit;

[0198] The point cloud processing module is configured to acquire the processed point cloud data according to the point cloud position information of each grid unit.

[0199] Figure 8 is a structural schematic diagram of a parking device provided by an embodiment of the present application, as Figure 8 shown, the apparatus 800 includes:

[0200] The point cloud acquisition module 801 is configured to acquire the point cloud data corresponding to the pre-acquired parking image in response to a parking trigger signal of a vehicle, the point cloud data being acquired according to the transformation matrix of the camera pose of the vehicle, the depth information of the obstacle detection point being acquired according to the multi-frame image tracking result of the obstacle detection point detected by the edge detection model, and the point cloud data being determined based on the depth information;

[0201] The recognition module 802 is configured to recognize the static obstacle in the parking scene of the vehicle according to the point cloud data.

[0202] The parking module 803 is configured to control a parking system of the vehicle to park according to the identification result of the static obstacle.

[0203] The embodiment of the present application also provides an electronic device, which comprises a processor 901 and a memory 902 connected with the processor 901. Figure 9 As shown in the figure, the electronic device comprises a processor 901 and a memory 902 connected with the processor 901.

[0204] The memory 902 stores computer-executed instructions.

[0205] The processor 901 executes the computer-executed instructions stored in the memory 902 to implement the point cloud extraction method based on a parking scene or the parking method.

[0206] The related description can be understood by referring to the related description and effects corresponding to the steps in the method embodiment of the present application, and will not be repeated here.

[0207] The embodiment of the present application also provides a computer readable storage medium, which stores computer-executed instructions, and the computer-executed instructions are used to implement the point cloud extraction method based on a parking scene or the parking method provided by the method embodiment.

[0208] The related description can be understood by referring to the related description and effects corresponding to the steps in the method embodiment of the present application, and will not be repeated here.

[0209] The embodiment of the present application also provides a computer program product, which comprises a computer program, and the computer program is used to implement the point cloud extraction method based on a parking scene or the parking method provided by the method embodiment when executed by a processor.

[0210] The related description can be understood by referring to the related description and effects corresponding to the steps in the method embodiment of the present application, and will not be repeated here.

[0211] The embodiment of the present application also provides a chip, which comprises a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory to execute the point cloud extraction method based on a parking scene or the parking method of the method embodiment.

[0212] The related description can be understood by referring to the related description and effects corresponding to the steps in the method embodiment of the present application, and will not be repeated here.

[0213] The computer readable storage medium can be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disc, and optical data storage device, etc.

[0214] Figure 10 A schematic block diagram of an example server 1000 that can be used to implement embodiments of the present disclosure is shown. The server is intended to represent various forms of digital computers, such as servers, blades, cloud servers, and the like. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0215] As Figure 10 shown, the server 1000 includes a computing unit 1001 that can perform various suitable actions and processes in accordance with computer programs stored in a read-only memory (ROM) 1002 or loaded into a random access memory (RAM) 1003 from the storage unit 1008. Various programs and data required for the operation of the server 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other by a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0216] A plurality of components in the server 800 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the server 1000 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0217] The computing unit 1001 can be various general purpose and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs various methods and processes described above, such as the above-described point cloud extraction method based on a parking scenario. For example, in some embodiments, the above-described point cloud extraction method based on a parking scenario can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the server 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the above-described point cloud extraction method based on a parking scenario described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the above-described point cloud extraction method based on a parking scenario by any other suitable means, such as by means of firmware.

[0218] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0219] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0220] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include a lined- up electrical connection, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0221] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, and for example, the division of the modules is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, apparatuses or modules, and can be electrical, mechanical or other forms.

[0222] Other embodiments of this application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0223] It will be understood that the application is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the application is indicated by the appended claims, rather than by the description.

Claims

1. A point cloud extraction method based on parking scenarios, characterized in that, include: Based on the vehicle's wheel speed pulse data and IMU data, determine the transformation matrix of the vehicle's camera pose. Based on an edge detection model, edge detection is performed on pre-acquired parking images to obtain edge detection results; wherein, the parking images are images of static obstacles acquired when the vehicle is parked, and the edge detection results include obstacle detection points related to the static obstacles; Based on the transformation matrix of the camera pose, the multi-frame image tracking results of the obstacle detection point are obtained, and the depth information of the obstacle detection point is obtained based on the multi-frame image tracking results, so as to obtain the point cloud data corresponding to the parking image based on the depth information.

2. The method according to claim 1, characterized in that, The step of determining the camera pose transformation matrix based on the vehicle's wheel speed pulse data and IMU data includes: The linear velocity change of the vehicle is obtained based on the wheel speed pulse data. Based on IMU data, obtain the acceleration changes and / or angular velocity changes of the vehicle; The transformation matrix of the camera pose is determined by integrating the linear velocity change, the acceleration change, and / or the angular velocity change.

3. The method according to claim 2, characterized in that, The process of fusing the linear velocity change, the acceleration, and / or the angular velocity change to determine the camera pose transformation matrix includes: The linear velocity change, acceleration change and / or angular velocity change are fused using the Kalman filter algorithm to obtain the fused linear velocity change, acceleration change and / or angular velocity change. The fused linear velocity change, acceleration change, and / or angular velocity change are input into the vehicle kinematics model to generate an incremental transformation matrix describing the vehicle pose change. The transformation matrix of the camera pose is obtained based on the incremental transformation matrix.

4. The method according to any one of claims 1-3, characterized in that, The step of obtaining multi-frame image tracking results of the obstacle detection points based on the transformation matrix of the camera pose includes: The obstacle detection point is located at its first position in the parking image at the first moment. Based on the transformation matrix of the camera pose, the first position is determined to be located at the second position in the parking image at the second time point; Based on the matching detection point corresponding to the second position, obtain the multi-frame image tracking results of the obstacle detection point.

5. The method according to claim 4, characterized in that, Determining the second position in the parking image at the second time moment based on the transformation matrix of the camera pose includes: Based on the transformation matrix of the camera pose, the epipolar line of the first position in the parking image at the second time time is calculated; wherein, the epipolar line is a line formed by the set of corresponding similar positions of the first position in the parking image at the second time time; The second position is determined by using an epipolar search algorithm to match the detection points of the obstacle detection points with the epipolar search.

6. The method according to claim 5, characterized in that, The step of calculating the epipolar line of the first position in the parking image at the second time moment based on the transformation matrix of the camera pose includes: Based on the transformation matrix and camera intrinsic parameter matrix of the camera pose, a fundamental matrix for multiple frames of images is determined. The fundamental matrix is ​​used to describe the geometric relationship between the parking image at the first moment and the parking image at the first moment. Based on the fundamental matrix and the homogeneous coordinates of the first position, the epipolar line of the first position in the parking image at the second time moment is calculated.

7. The method according to any one of claims 1-3, 5, and 6, characterized in that, The step of obtaining the point cloud data corresponding to the parking image based on the depth information includes: Based on the depth information and the pixel coordinates of the obstacle detection points, the coordinates are transformed to three-dimensional coordinates in the camera coordinate system using the camera intrinsic parameter matrix. The three-dimensional coordinates in the camera coordinate system are transformed to three-dimensional coordinates in the world coordinate system using the camera extrinsic parameter matrix to obtain the point cloud data corresponding to the parking image.

8. The method according to any one of claims 1-3, 5, and 6, characterized in that, After obtaining the point cloud data corresponding to the parking image based on the depth information, the method further includes: Divide the grid into multiple grid cells; each grid cell has a predetermined size. Based on the three-dimensional coordinates of each point in the point cloud data, each point is assigned to a corresponding grid cell; For each grid cell, the point cloud location information is obtained based on the weighted average location of all points within the grid cell; Based on the point cloud location information of each grid cell, the processed point cloud data is obtained.

9. A parking method, characterized in that, include: In response to the parking trigger signal of the vehicle, the point cloud data corresponding to the pre-collected parking image is acquired. The point cloud data is obtained by acquiring the multi-frame image tracking results of obstacle detection points detected by the edge detection model based on the transformation matrix of the vehicle's camera pose, and then obtaining the depth information of the obstacle detection points based on the multi-frame image tracking results, and determining the data based on the depth information. Based on the point cloud data, identify static obstacles in the vehicle parking scenario; Based on the identification results of the static obstacles, the vehicle's parking system is controlled to park.

10. A point cloud extraction device based on a parking scenario, characterized in that, include: The matrix determination module is used to determine the transformation matrix of the camera pose based on the vehicle's wheel speed pulse data and IMU data. The edge detection module is used to perform edge detection on the pre-acquired parking image based on the edge detection model to obtain the edge detection result, which includes obstacle detection points; The depth acquisition module is used to acquire multi-frame image tracking results of the obstacle detection point based on the transformation matrix of the camera pose, and to acquire the depth information of the obstacle detection point based on the multi-frame image tracking results. The point cloud extraction module is used to obtain point cloud data corresponding to the parking image based on the depth information.

11. A parking device, characterized in that, include: The point cloud acquisition module is used to acquire point cloud data corresponding to the pre-collected parking image in response to the parking trigger signal of the vehicle. The point cloud data is obtained by acquiring the multi-frame image tracking results of obstacle detection points detected by the edge detection model based on the transformation matrix of the vehicle's camera pose, and then obtaining the depth information of the obstacle detection points based on the multi-frame image tracking results, and determining the data based on the depth information. The recognition module is used to identify static obstacles in the vehicle parking scenario based on the point cloud data. The parking module is used to control the vehicle's parking system to park the vehicle based on the identification results of the static obstacles.

12. An electronic device, characterized in that, include: Memory and processor; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the electronic device to perform the method of any one of claims 1-9.

13. A computer-readable storage medium / computer program product, characterized in that, include: Memory and processor; The memory stores computer-executable instructions; the computer-readable storage medium stores computer-executable instructions that, when executed by a processor, are used to implement the method as described in any one of claims 1-9; and / or The computer program product includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-9.

Images