Automatic parking method and device based on multi-modal sensor data fusion

Abstract

The application relates to an automatic parking method and device based on multi-modal sensor data fusion, and belongs to the technical field of automatic parking. The method comprises the following steps: collecting vehicle multi-side sensor data, performing confidence weight weighted fusion on a first type of geometric features, a second type of distance features and a third type of visual features, and obtaining a fused BEV feature map; the fused BEV feature map comprises obstacle existence probability, obstacle type label and obstacle height information of each grid; target parking spaces and surrounding obstacles are identified based on the fused BEV feature map, and a parking path from a current position to the target parking space and capable of avoiding surrounding obstacles is planned in combination with vehicle kinematics constraints, which is favorable for realizing avoidance of various types of obstacles, thereby improving the reliability, precision and adaptability of automatic parking in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of automatic parking technology, and in particular to an automatic parking method and apparatus based on multimodal sensor data fusion. Background Technology

[0002] With the acceleration of urbanization and the continuous growth of car ownership, the scarcity of parking space is becoming increasingly prominent, placing higher demands on the efficiency and safety of parking operations. Automated parking systems, as a key function of advanced driver assistance systems and autonomous driving technologies, aim to reduce driver workload, decrease parking accidents, and improve space utilization through environmental perception, path planning, and vehicle control. However, due to limitations in environmental perception and insufficient sensor fusion, existing technologies still face significant challenges in terms of reliability, accuracy, and adaptability in complex scenarios. Summary of the Invention

[0003] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an automatic parking method and device based on multimodal sensor data fusion, which solves the technical problem that the reliability, accuracy and adaptability of automatic parking in complex scenarios still face significant challenges in the prior art.

[0004] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0005] The first aspect of this invention provides an automatic parking method based on multimodal sensor data fusion.

[0006] The automatic parking method based on multimodal sensor data fusion proposed in this embodiment of the invention includes:

[0007] Acquire 3D point cloud data generated by short-range dTOF sensors located on the left, right and rear sides of the vehicle, distance data generated by multiple ultrasonic radars distributed around the vehicle body, and image data generated by multiple surround-view cameras;

[0008] Outlier filtering and ground segmentation are performed on the three-dimensional point cloud data generated by the short-range dTOF sensor to extract the first type of geometric features characterizing low-lying obstacles and / or suspended obstacles;

[0009] Multipath interference filtering is applied to the distance data generated by the multiple ultrasonic radars to extract the second type of distance features that characterize the outline of the obstacle.

[0010] Distortion correction and perspective transformation are performed on the surround-view camera images to extract a third type of visual features containing the semantics of parking space lines, drivable areas, and obstacles;

[0011] The first type of geometric features, the second type of distance features, and the third type of visual features are fused using confidence weights to obtain a fused BEV feature map; the fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid cell;

[0012] Based on the fused BEV feature map, the target parking space and surrounding obstacles are identified. Combined with vehicle kinematic constraints, a parking path is planned from the current position to the target parking space that can avoid the surrounding obstacles. Based on the parking path, automatic parking is performed.

[0013] In some instances, the confidence-weighted fusion of the first type of geometric features, the second type of distance features, and the third type of visual features to obtain a fused BEV feature map includes:

[0014] The first type of geometric features, the second type of distance features, and the third type of visual features are uniformly transformed to a vehicle coordinate system with the rear axle center as the origin, to obtain the first type of geometric features, the second type of distance features, and the third type of visual features under a unified coordinate system.

[0015] The first type of geometric features, the second type of distance features, and the third type of visual features under unified coordinates are mapped to the same two-dimensional top-view plane to generate an initial BEV feature map.

[0016] Confidence weights are assigned to the dTOF sensor, the ultrasonic radar, and the surround-view camera, respectively, and are dynamically adjusted based on sensor characteristics and real-time environmental conditions.

[0017] Based on the confidence weights assigned to the dTOF sensor, the ultrasonic radar, and the surround-view camera, heterogeneous features at the same spatial location in the initial BEV feature map are fused to generate the fused BEV feature map.

[0018] In some instances, the identification of target parking spaces and surrounding obstacles based on the fused BEV feature map includes:

[0019] Based on the obstacle type label and the obstacle height information, the surrounding obstacles are classified into low obstacles that can be crossed, regular obstacles that need to be bypassed, and suspended obstacles that need to be avoided.

[0020] In some instances, the outlier filtering and ground segmentation of the 3D point cloud data generated by the short-range dTOF sensor, and the extraction of first-class geometric features characterizing low-lying obstacles and / or suspended obstacles, include:

[0021] Outlier points are filtered out from the three-dimensional point cloud data generated by the short-range dTOF sensor to obtain the point cloud after outlier removal.

[0022] Points in the point cloud after outlier filtering that are less than a first preset distance threshold from the fitting plane are marked as ground points, and these ground points are removed from the point cloud after outlier filtering to obtain a non-ground point cloud; wherein, the fitting plane is obtained by fitting using a random sampling consensus algorithm;

[0023] The non-ground point cloud is clustered at a height to obtain candidate low-lying obstacle cloud clusters with a maximum height lower than a first height threshold;

[0024] Low obstacle cloud clusters with both density and area greater than the corresponding thresholds are selected from the candidate low obstacle cloud clusters, and the outline polygons of the low obstacle cloud clusters are extracted as low obstacle features in the first type of geometric features.

[0025] In some instances, the outlier filtering and ground segmentation of the 3D point cloud data generated by the short-range dTOF sensor, and the extraction of first-class geometric features characterizing low-lying obstacles and / or suspended obstacles, include:

[0026] Outlier points are filtered out from the three-dimensional point cloud data generated by the short-range dTOF sensor to obtain the point cloud after outlier removal.

[0027] Points in the point cloud after outlier filtering that are less than a first preset distance threshold from the fitting plane are marked as ground points, and these ground points are removed from the point cloud after outlier filtering to obtain a non-ground point cloud; wherein, the fitting plane is obtained by fitting using a random sampling consensus algorithm;

[0028] The non-ground point cloud is subjected to height clustering to obtain candidate suspended obstacle point cloud clusters with a ground height greater than a second height threshold; wherein, the second height threshold is greater than the first height threshold;

[0029] In the candidate suspended obstacle point cloud clusters, suspended obstacle point cloud clusters with a lower support point cloud density below a preset support density threshold are selected, and the three-dimensional bounding box and the lowest point height of the suspended obstacle point cloud clusters are extracted as suspended obstacle features in the first type of geometric features.

[0030] In some instances, the step of performing multipath interference filtering on the distance data generated by the plurality of ultrasonic radars to extract a second type of distance feature characterizing the obstacle contour includes:

[0031] Obtain the echo width and signal attenuation characteristics of the raw echo signal for each ultrasonic radar channel;

[0032] Based on the echo width and signal attenuation characteristics of the original echo signal of each ultrasonic radar channel, suspected multipath interference data in the distance data is determined;

[0033] By performing coordinate matching between the suspected multipath interference data and the three-dimensional point cloud data, multipath interference data is identified from the suspected multipath interference data, and the multipath interference data is removed from the distance data to obtain the filtered effective ultrasonic distance data.

[0034] Spatial clustering is performed on the filtered effective ultrasonic distance data to aggregate measurement points from different radars that point to similar spatial locations into the same candidate obstacle, and the geometric center, contour boundary points and distance variance of each candidate obstacle are obtained;

[0035] Based on the geometric center, contour boundary points, and distance variance of each candidate obstacle, a polygonal obstacle contour is obtained, and the polygonal obstacle contour is used as the second type of distance feature.

[0036] In some instances, the distortion correction and perspective transformation of the surround-view camera image, and the extraction of a third type of visual features containing the semantics of parking space lines, drivable areas, and obstacles, include:

[0037] The distortion of the surround-view camera images is corrected to obtain four distortion-free images.

[0038] Based on the preset perspective transformation matrix, the four images are projected onto a unified bird's-eye view plane to generate a panoramic top-down stitched image.

[0039] The panoramic top-down stitched image is input into a pre-trained multi-task convolutional neural network to obtain a parking space line segmentation map, a drivable area segmentation map, and an obstacle semantic segmentation map.

[0040] The parking space line segmentation map, the drivable area segmentation map, and the obstacle semantic segmentation map are subjected to geometric feature and semantic information fusion processing to obtain the third type of visual features.

[0041] A second aspect of this invention provides an automatic parking device based on multimodal sensor data fusion, comprising:

[0042] The data acquisition unit is used to acquire three-dimensional point cloud data generated by short-range dTOF sensors located on the left, right and rear sides of the vehicle, distance data generated by multiple ultrasonic radars distributed around the vehicle body, and image data generated by multiple surround-view cameras.

[0043] The first type of geometric feature extraction unit is used to filter out outliers and segment the ground in the three-dimensional point cloud data generated by the short-range dTOF sensor, and extract the first type of geometric features that characterize low obstacles and / or suspended obstacles.

[0044] The second type of distance feature extraction unit is used to perform multipath interference filtering on the distance data generated by the multiple ultrasonic radars and extract the second type of distance features that characterize the outline of the obstacle.

[0045] The third type of visual feature extraction unit is used to perform distortion correction and perspective transformation on the surround view camera image and extract the third type of visual features containing the semantics of parking space lines, drivable areas and obstacles.

[0046] The feature fusion unit is used to perform confidence-weighted fusion of the first type of geometric features, the second type of distance features, and the third type of visual features to obtain a fused BEV feature map; the fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid cell;

[0047] The automatic parking unit is used to identify the target parking space and surrounding obstacles based on the fused BEV feature map, and, in combination with vehicle kinematic constraints, plan a parking path from the current position to the target parking space that can avoid the surrounding obstacles, and perform automatic parking based on the parking path.

[0048] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in the first aspect above.

[0049] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect above.

[0050] An automatic parking method based on multimodal sensor data fusion of the present invention includes: acquiring three-dimensional point cloud data generated by short-range dTOF sensors located on the left, right, and rear sides of the vehicle; distance data generated by multiple ultrasonic radars distributed around the vehicle body; and image data generated by multiple surround-view cameras; performing outlier filtering and ground segmentation on the three-dimensional point cloud data generated by the short-range dTOF sensors to extract first-type geometric features representing low-lying obstacles and / or suspended obstacles; performing multipath interference filtering on the distance data generated by the multiple ultrasonic radars to extract second-type distance features representing obstacle contours; and performing multipath interference filtering on the distance data generated by the surround-view cameras to extract second-type distance features representing obstacle contours; and performing multipath interference filtering on the distance data generated by the multiple ultrasonic radars ... The camera image undergoes distortion correction and perspective transformation to extract a third type of visual feature containing the semantics of parking space lines, drivable areas, and obstacles. The first type of geometric feature, the second type of distance feature, and the third type of visual feature are then fused using confidence weights to obtain a fused BEV feature map. The fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid cell. Based on the fused BEV feature map, the target parking space and surrounding obstacles are identified. Combined with vehicle kinematic constraints, a parking path is planned from the current position to the target parking space, avoiding surrounding obstacles. Automatic parking is then performed based on this parking path. This application acquires data from multiple vehicle sensors, performs confidence-weighted fusion of first-type geometric features, second-type distance features, and third-type visual features to obtain a fused BEV feature map. The fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid cell. Based on the fused BEV feature map, the target parking space and surrounding obstacles are identified. Combined with vehicle kinematic constraints, a parking path is planned from the current position to the target parking space while avoiding surrounding obstacles. This facilitates the avoidance of various types of obstacles, thereby improving the reliability, accuracy, and adaptability of automatic parking in complex scenarios. Attached Figure Description

[0051] Figure 1 A flowchart of an automatic parking method based on multimodal sensor data fusion is provided for an embodiment of the present invention;

[0052] Figure 2 An automatic parking logic block diagram based on multimodal sensor data fusion is provided for an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of an automatic parking device based on multimodal sensor data fusion, provided as an embodiment of the present invention. Detailed Implementation

[0054] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] The automatic parking method based on multimodal sensor data fusion proposed in this invention addresses the significant challenges to the reliability, accuracy, and adaptability of automatic parking in complex scenarios in existing technologies. By collecting data from multiple vehicle sensors, the method performs confidence-weighted fusion of first-type geometric features, second-type distance features, and third-type visual features to obtain a fused BEV feature map. The fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid cell. Based on the fused BEV feature map, the method identifies the target parking space and surrounding obstacles. Combined with vehicle kinematic constraints, it plans a parking path from the current position to the target parking space that avoids surrounding obstacles, facilitating the avoidance of various obstacle types and thus improving the reliability, accuracy, and adaptability of automatic parking in complex scenarios.

[0056] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0057] Figure 1 This is a flowchart illustrating an automatic parking method based on multimodal sensor data fusion, provided as an embodiment of the present invention. Figure 1 As shown in the embodiment of the present invention, the automatic parking method based on multimodal sensor data fusion includes:

[0058] Step 100: Acquire 3D point cloud data generated by short-range dTOF sensors located on the left, right and rear sides of the vehicle, distance data generated by multiple ultrasonic radars distributed around the vehicle body, and image data generated by multiple surround-view cameras.

[0059] Step 110: Filter out outliers and segment the ground in the three-dimensional point cloud data generated by the short-range dTOF sensor, and extract the first type of geometric features that characterize low-lying obstacles and / or suspended obstacles.

[0060] Step 120: Perform multipath interference filtering on the distance data generated by the multiple ultrasonic radars to extract the second type of distance features that characterize the outline of the obstacle;

[0061] Step 130: Perform distortion correction and perspective transformation on the surround view camera image, and extract the third type of visual features containing the semantics of parking space lines, drivable area and obstacles;

[0062] Step 140: Perform confidence-weighted fusion of the first type of geometric features, the second type of distance features, and the third type of visual features to obtain a fused BEV feature map; the fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid cell;

[0063] Step 150: Identify the target parking space and surrounding obstacles based on the fused BEV feature map, and combine vehicle kinematic constraints to plan a parking path from the current position to the target parking space that can avoid the surrounding obstacles, and perform automatic parking based on the parking path.

[0064] In this exemplary embodiment, the vehicle may employ three short-range dTOF sensors, twelve ultrasonic radars, four surround-view cameras, and an autonomous driving controller. The three short-range dTOF sensors, twelve ultrasonic radars, and four surround-view cameras are perception sensors responsible for obstacle detection and output, while the intelligent driving controller is responsible for perception fusion and generating a BEV bird's-eye view, then performing path planning to complete automatic parking.

[0065] The system includes three short-range dTOF sensors, one each on the left, right, and rear fenders; twelve ultrasonic radars, four directly in front of the vehicle, one on each of the left and right front sides, four directly behind the vehicle, and one on each of the left and right rear sides; and four surround-view cameras, one in front of the vehicle, one on the left exterior rearview mirror, one on the right exterior rearview mirror, and one directly behind the vehicle. dTOF (direct time-of-flight) is a laser imaging ranging sensor, a high-precision, high-speed non-contact measurement device. Short-range dTOF offers advantages such as simple structure, small size, high measurement accuracy, fast response speed, accurate simultaneous detection of multiple objects, and low power consumption. It emits laser pulses and receives the laser signals reflected back from the target object, calculating the distance to the target object using the speed of light and time difference to achieve high-precision ranging. Its working principle is based on the time-of-flight method, utilizing the linear propagation characteristics of the laser beam to ensure the accuracy and stability of the measurement results. Laser imaging rangefinders can not only measure the distance to objects, but also perform image scanning and 3D modeling, thereby identifying small objects around a vehicle and scanning them into 3D models.

[0066] This application acquires data from multiple vehicle sensors, performs confidence-weighted fusion of first-type geometric features, second-type distance features, and third-type visual features to obtain a fused BEV feature map. The fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid cell. Based on the fused BEV feature map, the target parking space and surrounding obstacles are identified. Combined with vehicle kinematic constraints, a parking path is planned from the current position to the target parking space, avoiding surrounding obstacles. This facilitates the avoidance of various obstacle types, thereby improving the reliability, accuracy, and adaptability of automatic parking in complex scenarios. By fusing 3D TOF, ultrasonic radar, and camera data, and generating a bird's-eye view using the BEV algorithm, small and irregular obstacles around the vehicle can be accurately identified, improving obstacle avoidance capabilities and thus enhancing automatic parking performance.

[0067] Figure 2 This is a logic block diagram of automatic parking based on multimodal sensor data fusion, provided as an embodiment of the present invention. For example... Figure 2 As shown, the vehicle perception system 20 includes ultrasonic radar, a surround-view camera, and a dTOF sensor. The autonomous driving controller 21 is used for path planning based on the BEV bird's-eye view and algorithm control data. The vehicle attitude controller 22 is used to control the vehicle's steering, driving, and braking according to the path planning.

[0068] In some instances, the confidence-weighted fusion of the first type of geometric features, the second type of distance features, and the third type of visual features to obtain a fused BEV feature map includes:

[0069] The first type of geometric features, the second type of distance features, and the third type of visual features are uniformly transformed to a vehicle coordinate system with the rear axle center as the origin, to obtain the first type of geometric features, the second type of distance features, and the third type of visual features under a unified coordinate system.

[0070] The first type of geometric features, the second type of distance features, and the third type of visual features under unified coordinates are mapped to the same two-dimensional top-view plane to generate an initial BEV feature map.

[0071] Confidence weights are assigned to the dTOF sensor, the ultrasonic radar, and the surround-view camera, respectively, and are dynamically adjusted based on sensor characteristics and real-time environmental conditions.

[0072] Based on the confidence weights assigned to the dTOF sensor, the ultrasonic radar, and the surround-view camera, heterogeneous features at the same spatial location in the initial BEV feature map are fused to generate the fused BEV feature map.

[0073] In this exemplary embodiment, based on the confidence weights assigned to the dTOF sensor, the ultrasonic radar, and the surround-view camera, the heterogeneous features at the same spatial location in the initial BEV feature map are weighted and fused to generate the fused BEV feature map.

[0074] In this exemplary embodiment, the dynamic adjustment of the confidence weight is based at least in part on one or a combination of the following factors:

[0075] The signal-to-noise ratio of the dTOF sensor under the current light intensity;

[0076] The visibility and texture richness of the surround-view camera image in the current environment;

[0077] The degree of consistency between the ultrasonic radar data and the dTOF point cloud data within the overlapping detection area.

[0078] In this exemplary embodiment, the step of identifying the target parking space and surrounding obstacles based on the fused BEV feature map, and planning a parking path from the current position to the target parking space that can avoid the surrounding obstacles in conjunction with vehicle kinematic constraints, includes:

[0079] If the obstacle is identified as a low obstacle with a height lower than the minimum ground clearance of the vehicle chassis and a stable structure, its avoidance priority is reduced in the trajectory cost function, allowing the trajectory to approach or plan a brief crossing.

[0080] If it is identified as a suspended obstacle, its vertical projection area is constructed as a no-entry zone in the fused BEV feature map, and the planned trajectory must detour around it;

[0081] For conventional obstacles, the method of expanding their contours and adding trajectory optimization constraints is used for obstacle avoidance.

[0082] In this exemplary embodiment, the step of identifying the target parking space and surrounding obstacles based on the fused BEV feature map includes:

[0083] Based on the obstacle type label and the obstacle height information, the surrounding obstacles are classified into low obstacles that can be crossed, regular obstacles that need to be bypassed, and suspended obstacles that need to be avoided.

[0084] In some instances, the outlier filtering and ground segmentation of the 3D point cloud data generated by the short-range dTOF sensor, and the extraction of first-class geometric features characterizing low-lying obstacles and / or suspended obstacles, include:

[0085] Outlier points are filtered out from the three-dimensional point cloud data generated by the short-range dTOF sensor to obtain the point cloud after outlier removal.

[0086] Points in the point cloud after outlier filtering that are less than a first preset distance threshold from the fitting plane are marked as ground points, and these ground points are removed from the point cloud after outlier filtering to obtain a non-ground point cloud; wherein, the fitting plane is obtained by fitting using a random sampling consensus algorithm;

[0087] The non-ground point cloud is clustered at a height to obtain candidate low-lying obstacle cloud clusters with a maximum height lower than a first height threshold;

[0088] Low obstacle cloud clusters with both density and area greater than the corresponding thresholds are selected from the candidate low obstacle cloud clusters, and the outline polygons of the low obstacle cloud clusters are extracted as low obstacle features in the first type of geometric features.

[0089] In this exemplary embodiment, for the original three-dimensional point cloud data, a statistical outlier removal algorithm is first used to calculate the average distance of the K nearest neighbors of each point and remove outliers whose distance distribution exceeds the preset standard deviation range.

[0090] For the point cloud after filtering out outliers, a random sampling consensus algorithm is used to fit a ground plane model, which is represented by the plane equation ax + by + cz + d = 0. Points whose distance from the fitted plane is less than a first preset distance threshold are marked as ground points and removed to obtain a non-ground point cloud.

[0091] The non-ground point cloud is clustered based on height, and point cloud clusters that are continuous and whose maximum height is lower than a first height threshold are identified as candidate low-lying obstacles.

[0092] Calculate the point cloud density and horizontal projected area of ​​each candidate cluster, filter out clusters whose density and area are both greater than the corresponding threshold, and extract their outline polygons as low-lying obstacle features in the first type of geometric features.

[0093] In some instances, the outlier filtering and ground segmentation of the 3D point cloud data generated by the short-range dTOF sensor, and the extraction of first-class geometric features characterizing low-lying obstacles and / or suspended obstacles, include:

[0094] Outlier points are filtered out from the three-dimensional point cloud data generated by the short-range dTOF sensor to obtain the point cloud after outlier removal.

[0095] Points in the point cloud after outlier filtering that are less than a first preset distance threshold from the fitting plane are marked as ground points, and these ground points are removed from the point cloud after outlier filtering to obtain a non-ground point cloud; wherein, the fitting plane is obtained by fitting using a random sampling consensus algorithm;

[0096] The non-ground point cloud is subjected to height clustering to obtain candidate suspended obstacle point cloud clusters with a ground height greater than a second height threshold; wherein, the second height threshold is greater than the first height threshold;

[0097] In the candidate suspended obstacle point cloud clusters, suspended obstacle point cloud clusters with a lower support point cloud density below a preset support density threshold are selected, and the three-dimensional bounding box and the lowest point height of the suspended obstacle point cloud clusters are extracted as suspended obstacle features in the first type of geometric features.

[0098] In this exemplary embodiment, point cloud clusters with a ground elevation greater than a second elevation threshold are identified as candidate suspended obstacles in the non-ground point cloud.

[0099] For each candidate cluster, search downwards along the vertical direction to determine whether there are sufficient support point clouds within its vertical projection region;

[0100] If the support point cloud density is lower than the preset support density threshold, the candidate cluster is determined to be a suspended obstacle, and its three-dimensional bounding box and the height of the lowest point are extracted as suspended obstacle features in the first type of geometric features.

[0101] In some instances, the step of performing multipath interference filtering on the distance data generated by the plurality of ultrasonic radars to extract a second type of distance feature characterizing the obstacle contour includes:

[0102] Obtain the echo width and signal attenuation characteristics of the raw echo signal for each ultrasonic radar channel;

[0103] Based on the echo width and signal attenuation characteristics of the original echo signal of each ultrasonic radar channel, suspected multipath interference data in the distance data is determined;

[0104] By performing coordinate matching between the suspected multipath interference data and the three-dimensional point cloud data, multipath interference data is identified from the suspected multipath interference data, and the multipath interference data is removed from the distance data to obtain the filtered effective ultrasonic distance data.

[0105] Spatial clustering is performed on the filtered effective ultrasonic distance data to aggregate measurement points from different radars that point to similar spatial locations into the same candidate obstacle, and the geometric center, contour boundary points and distance variance of each candidate obstacle are obtained;

[0106] Based on the geometric center, contour boundary points, and distance variance of each candidate obstacle, a polygonal obstacle contour is obtained, and the polygonal obstacle contour is used as the second type of distance feature.

[0107] In this exemplary embodiment, for the raw echo signal acquired by each ultrasonic radar channel, its echo width and signal attenuation characteristics are calculated;

[0108] When the detected echo width exceeds the expected width threshold for a single reflection, and / or the signal attenuation pattern conforms to the characteristics of multiple reflections, the measurement value is marked as suspected multipath interference.

[0109] The measurement data of each ultrasonic radar is matched with the spatiotemporally aligned dTOF point cloud data.

[0110] For the measurements of the suspected multipath interference, search for valid dTOF point clouds within the corresponding spatial region:

[0111] If there is a dense and continuous dTOF obstacle point cloud in the area, and its measurement distance is less than the ultrasonic measurement distance, then the ultrasonic data is confirmed to be multipath interference and should be removed or significantly downweighted.

[0112] If the dTOF point cloud in the region is sparse or has no valid data, the ultrasonic data is retained, but its confidence level is reduced.

[0113] Spatial clustering is performed on the filtered effective ultrasonic distance data to aggregate measurement points from different radars that point to similar spatial locations into the same candidate obstacle;

[0114] For each candidate obstacle, calculate its geometric center, contour boundary points, and distance variance to form a preliminary obstacle contour description;

[0115] The preliminary contour is verified and optimized by combining the semantic segmentation results of the surround view camera; if the camera identifies a clear obstacle edge in the corresponding area, the boundary of the ultrasonic contour is corrected with the visual edge as a reference.

[0116] For areas that are not clearly visible but can be continuously and stably detected by ultrasound, retain and enhance the ultrasound contour features.

[0117] The optimized obstacle contour polygon is output as the second type of distance feature.

[0118] In some instances, the distortion correction and perspective transformation of the surround-view camera image, and the extraction of a third type of visual features containing the semantics of parking space lines, drivable areas, and obstacles, include:

[0119] The distortion of the surround-view camera images is corrected to obtain four distortion-free images.

[0120] Based on the preset perspective transformation matrix, the four images are projected onto a unified bird's-eye view plane to generate a panoramic top-down stitched image.

[0121] The panoramic top-down stitched image is input into a pre-trained multi-task convolutional neural network to obtain a parking space line segmentation map, a drivable area segmentation map, and an obstacle semantic segmentation map.

[0122] The parking space line segmentation map, the drivable area segmentation map, and the obstacle semantic segmentation map are subjected to geometric feature and semantic information fusion processing to obtain the third type of visual features.

[0123] In this exemplary embodiment, the input raw fisheye image is subjected to distortion correction processing based on the pre-calibrated intrinsic parameters and distortion coefficients of each surround-view camera;

[0124] The four distortion-free images are projected onto a unified bird's-eye view plane based on their extrinsic parameters and a preset perspective transformation matrix to generate a panoramic top-down stitched image.

[0125] The panoramic top-down stitched image is input into a trained multi-task convolutional neural network to obtain a parking space line segmentation map, a drivable area segmentation map, and an obstacle semantic segmentation map. In the parking space line segmentation map, each pixel is labeled as a parking space line, a parking space corner, or the background; the drivable area segmentation map can distinguish between drivable road surfaces, non-drivable areas, and unknown areas; and the obstacle semantic segmentation map can identify pixel-level categories of vehicles, pedestrians, cones, and posts.

[0126] Then, corner detection and line fitting are performed on the parking space line segmentation map to generate a set of candidate parking space polygons defined by corner coordinates and connection relationships;

[0127] By overlaying and analyzing the drivable area segmentation map and the obstacle semantic segmentation map, drivable areas occupied by obstacles are identified.

[0128] The output consists of structured data containing the following information, which is presented as the third type of visual feature. This information includes:

[0129] A list of candidate parking spaces, each containing the coordinates of its polygon vertices and its type (parallel / perpendicular / diagonal).

[0130] Driveable area mask;

[0131] A list of obstacles, each containing its pixel mask, semantic category, and estimated position in the bird's-eye view coordinate system.

[0132] This invention provides an automatic parking device based on multimodal sensor data fusion. Figure 3 This is a schematic diagram of an automatic parking device based on multimodal sensor data fusion, provided as an embodiment of the present invention. Figure 3 As shown, the automatic parking device based on multimodal sensor data fusion includes:

[0133] The data acquisition unit 30 is used to acquire three-dimensional point cloud data generated by short-range dTOF sensors located on the left, right and rear sides of the vehicle, distance data generated by multiple ultrasonic radars distributed around the vehicle body, and image data generated by multiple surround-view cameras.

[0134] The first type of geometric feature extraction unit 31 is used to filter out outliers and segment the ground in the three-dimensional point cloud data generated by the short-range dTOF sensor, and extract the first type of geometric features that characterize low obstacles and / or suspended obstacles.

[0135] The second type of distance feature extraction unit 32 is used to perform multipath interference filtering on the distance data generated by the multiple ultrasonic radars and extract the second type of distance features that characterize the outline of the obstacle.

[0136] The third type of visual feature extraction unit 33 is used to perform distortion correction and perspective transformation on the surround view camera image and extract the third type of visual features including parking space lines, drivable area and obstacle semantics.

[0137] The feature fusion unit 34 is used to perform confidence weighted fusion of the first type of geometric features, the second type of distance features, and the third type of visual features to obtain a fused BEV feature map; the fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid.

[0138] The automatic parking unit 35 is used to identify the target parking space and surrounding obstacles based on the fused BEV feature map, and, in combination with vehicle kinematic constraints, plan a parking path from the current position to the target parking space that can avoid the surrounding obstacles, and perform automatic parking based on the parking path.

[0139] In this exemplary embodiment, the vehicle may employ three short-range dTOF sensors, twelve ultrasonic radars, four surround-view cameras, and an autonomous driving controller. The three short-range dTOF sensors, twelve ultrasonic radars, and four surround-view cameras are perception sensors responsible for obstacle detection and output, while the intelligent driving controller is responsible for perception fusion and generating a BEV bird's-eye view, then performing path planning to complete automatic parking.

[0140] The system includes three short-range dTOF sensors, one each on the left, right, and rear fenders; twelve ultrasonic radars, four directly in front of the vehicle, one on each of the left and right front sides, four directly behind the vehicle, and one on each of the left and right rear sides; and four surround-view cameras, one in front of the vehicle, one on the left exterior rearview mirror, one on the right exterior rearview mirror, and one directly behind the vehicle. dTOF (direct time-of-flight) is a laser imaging ranging sensor, a high-precision, high-speed non-contact measurement device. Short-range dTOF offers advantages such as simple structure, small size, high measurement accuracy, fast response speed, accurate simultaneous detection of multiple objects, and low power consumption. It emits laser pulses and receives the laser signals reflected back from the target object, calculating the distance to the target object using the speed of light and time difference to achieve high-precision ranging. Its working principle is based on the time-of-flight method, utilizing the linear propagation characteristics of the laser beam to ensure the accuracy and stability of the measurement results. Laser imaging rangefinders can not only measure the distance to objects, but also perform image scanning and 3D modeling, thereby identifying small objects around a vehicle and scanning them into 3D models.

[0141] This application acquires data from multiple vehicle sensors, performs confidence-weighted fusion of first-type geometric features, second-type distance features, and third-type visual features to obtain a fused BEV feature map. The fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid cell. Based on the fused BEV feature map, the target parking space and surrounding obstacles are identified. Combined with vehicle kinematic constraints, a parking path is planned from the current position to the target parking space, avoiding surrounding obstacles. This facilitates the avoidance of various obstacle types, thereby improving the reliability, accuracy, and adaptability of automatic parking in complex scenarios. By fusing 3D TOF, ultrasonic radar, and camera data, and generating a bird's-eye view using the BEV algorithm, small and irregular obstacles around the vehicle can be accurately identified, improving obstacle avoidance capabilities and thus enhancing automatic parking performance.

[0142] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0143] This invention provides a computer-readable storage medium, characterized in that it stores an automatic parking program based on multimodal sensor data fusion. When the automatic parking program based on multimodal sensor data fusion is executed by a processor, it implements the automatic parking method based on multimodal sensor data fusion described in the above embodiments.

[0144] This invention provides an electronic device, characterized in that it includes a memory, a processor, and an automatic parking program based on multimodal sensor data fusion stored in the memory and executable on the processor. When the processor executes the automatic parking program based on multimodal sensor data fusion, it implements the automatic parking method based on multimodal sensor data fusion described in the above embodiments.

[0145] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0146] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0147] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that they are in indirect contact through an intermediate medium. Furthermore, "above," "over," or "on top" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," or "beneath" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0148] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0149] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An automatic parking method based on multimodal sensor data fusion, characterized in that, include: Acquire 3D point cloud data generated by short-range dTOF sensors located on the left, right and rear sides of the vehicle, distance data generated by multiple ultrasonic radars distributed around the vehicle body, and image data generated by multiple surround-view cameras; Outlier filtering and ground segmentation are performed on the three-dimensional point cloud data generated by the short-range dTOF sensor to extract the first type of geometric features characterizing low-lying obstacles and / or suspended obstacles; Multipath interference filtering is applied to the distance data generated by the multiple ultrasonic radars to extract the second type of distance features that characterize the outline of the obstacle. Distortion correction and perspective transformation are performed on the surround-view camera images to extract a third type of visual features containing the semantics of parking space lines, drivable areas, and obstacles; The first type of geometric features, the second type of distance features, and the third type of visual features are fused using confidence weights to obtain a fused BEV feature map; the fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid cell; Based on the fused BEV feature map, the target parking space and surrounding obstacles are identified. Combined with vehicle kinematic constraints, a parking path is planned from the current position to the target parking space that can avoid the surrounding obstacles. Based on the parking path, automatic parking is performed.

2. The automatic parking method based on multimodal sensor data fusion according to claim 1, characterized in that, The step of performing confidence-weighted fusion of the first type of geometric features, the second type of distance features, and the third type of visual features to obtain a fused BEV feature map includes: The first type of geometric features, the second type of distance features, and the third type of visual features are uniformly transformed to a vehicle coordinate system with the rear axle center as the origin, to obtain the first type of geometric features, the second type of distance features, and the third type of visual features under a unified coordinate system. The first type of geometric features, the second type of distance features, and the third type of visual features under unified coordinates are mapped to the same two-dimensional top-view plane to generate an initial BEV feature map. Confidence weights are assigned to the dTOF sensor, the ultrasonic radar, and the surround-view camera, respectively, and are dynamically adjusted based on sensor characteristics and real-time environmental conditions. Based on the confidence weights assigned to the dTOF sensor, the ultrasonic radar, and the surround-view camera, heterogeneous features at the same spatial location in the initial BEV feature map are fused to generate the fused BEV feature map.

3. The automatic parking method based on multimodal sensor data fusion according to claim 1, characterized in that, The identification of the target parking space and surrounding obstacles based on the fused BEV feature map includes: Based on the obstacle type label and the obstacle height information, the surrounding obstacles are classified into low obstacles that can be crossed, regular obstacles that need to be bypassed, and suspended obstacles that need to be avoided.

4. The automatic parking method based on multimodal sensor data fusion according to claim 1, characterized in that, The process of filtering out outliers and segmenting the ground in the 3D point cloud data generated by the short-range dTOF sensor to extract first-class geometric features characterizing low-lying obstacles and / or suspended obstacles includes: Outlier points are filtered out from the three-dimensional point cloud data generated by the short-range dTOF sensor to obtain the point cloud after outlier removal. Points in the point cloud after outlier filtering that are less than a first preset distance threshold from the fitting plane are marked as ground points, and these ground points are removed from the point cloud after outlier filtering to obtain a non-ground point cloud; wherein, the fitting plane is obtained by fitting using a random sampling consensus algorithm; The non-ground point cloud is clustered at a height to obtain candidate low-lying obstacle cloud clusters with a maximum height lower than a first height threshold; Low obstacle cloud clusters with both density and area greater than the corresponding thresholds are selected from the candidate low obstacle cloud clusters, and the outline polygons of the low obstacle cloud clusters are extracted as low obstacle features in the first type of geometric features.

5. The automatic parking method based on multimodal sensor data fusion according to claim 4, characterized in that, The process of filtering out outliers and segmenting the ground in the 3D point cloud data generated by the short-range dTOF sensor to extract first-class geometric features characterizing low-lying obstacles and / or suspended obstacles includes: Outlier points are filtered out from the three-dimensional point cloud data generated by the short-range dTOF sensor to obtain the point cloud after outlier removal. Points in the point cloud after outlier filtering that are less than a first preset distance threshold from the fitting plane are marked as ground points, and these ground points are removed from the point cloud after outlier filtering to obtain a non-ground point cloud; wherein, the fitting plane is obtained by fitting using a random sampling consensus algorithm; The non-ground point cloud is subjected to height clustering to obtain candidate suspended obstacle point cloud clusters with a ground height greater than a second height threshold; wherein, the second height threshold is greater than the first height threshold; In the candidate suspended obstacle point cloud clusters, suspended obstacle point cloud clusters with a lower support point cloud density below a preset support density threshold are selected, and the three-dimensional bounding box and the lowest point height of the suspended obstacle point cloud clusters are extracted as suspended obstacle features in the first type of geometric features.

6. The automatic parking method based on multimodal sensor data fusion according to claim 1, characterized in that, The step of performing multipath interference filtering on the distance data generated by the multiple ultrasonic radars to extract a second type of distance feature representing the obstacle contour includes: Obtain the echo width and signal attenuation characteristics of the raw echo signal for each ultrasonic radar channel; Based on the echo width and signal attenuation characteristics of the original echo signal of each ultrasonic radar channel, suspected multipath interference data in the distance data is determined; By performing coordinate matching between the suspected multipath interference data and the three-dimensional point cloud data, multipath interference data is identified from the suspected multipath interference data, and the multipath interference data is removed from the distance data to obtain the filtered effective ultrasonic distance data. Spatial clustering is performed on the filtered effective ultrasonic distance data to aggregate measurement points from different radars that point to similar spatial locations into the same candidate obstacle, and the geometric center, contour boundary points and distance variance of each candidate obstacle are obtained; Based on the geometric center, contour boundary points, and distance variance of each candidate obstacle, a polygonal obstacle contour is obtained, and the polygonal obstacle contour is used as the second type of distance feature.

7. The automatic parking method based on multimodal sensor data fusion according to claim 1, characterized in that, The distortion correction and perspective transformation of the surround-view camera images are performed to extract a third type of visual feature containing the semantics of parking space lines, drivable areas, and obstacles, including: The distortion of the surround-view camera images is corrected to obtain four distortion-free images. Based on the preset perspective transformation matrix, the four images are projected onto a unified bird's-eye view plane to generate a panoramic top-down stitched image. The panoramic top-down stitched image is input into a pre-trained multi-task convolutional neural network to obtain a parking space line segmentation map, a drivable area segmentation map, and an obstacle semantic segmentation map. The parking space line segmentation map, the drivable area segmentation map, and the obstacle semantic segmentation map are subjected to geometric feature and semantic information fusion processing to obtain the third type of visual features.

8. An automatic parking device based on multimodal sensor data fusion, characterized in that, include: The data acquisition unit is used to acquire three-dimensional point cloud data generated by short-range dTOF sensors located on the left, right and rear sides of the vehicle, distance data generated by multiple ultrasonic radars distributed around the vehicle body, and image data generated by multiple surround-view cameras. The first type of geometric feature extraction unit is used to filter out outliers and segment the ground in the three-dimensional point cloud data generated by the short-range dTOF sensor, and extract the first type of geometric features that characterize low obstacles and / or suspended obstacles. The second type of distance feature extraction unit is used to perform multipath interference filtering on the distance data generated by the multiple ultrasonic radars and extract the second type of distance features that characterize the outline of the obstacle. The third type of visual feature extraction unit is used to perform distortion correction and perspective transformation on the surround view camera image and extract the third type of visual features containing the semantics of parking space lines, drivable areas and obstacles. The feature fusion unit is used to perform confidence-weighted fusion of the first type of geometric features, the second type of distance features, and the third type of visual features to obtain a fused BEV feature map; the fused BEV feature map includes the obstacle presence probability, obstacle type label, and obstacle height information for each grid cell; The automatic parking unit is used to identify the target parking space and surrounding obstacles based on the fused BEV feature map, and, in combination with vehicle kinematic constraints, plan a parking path from the current position to the target parking space that can avoid the surrounding obstacles, and perform automatic parking based on the parking path.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

Images