Automatic parking method and system capable of sensing three-dimensional protruding obstacles

Abstract

The invention provides an automatic parking method and system capable of perceiving a three-dimensional protruding obstacle, and belongs to the technical field of automatic parking, and the method comprises the steps: obtaining an empty parking space, a driving space and a conventional obstacle based on a vision and ultrasonic radar sensor; using a pre-trained three-dimensional panoramic instance segmentation occupation network model to obtain a three-dimensional panoramic instance segmentation result of the surrounding environment of the own vehicle including the three-dimensional protruding obstacles; constructing a vehicle surface dense voxel of the own vehicle; the three-dimensional protruding obstacles matched with the empty parking spaces are added to a matched three-dimensional protruding obstacle list of the empty parking spaces; selecting a parking key position; calculating the 3D space minimum distance between the target three-dimensional protruding obstacle and the surface dense voxels of the side surface corresponding to the parking key position of the target empty parking space; when the minimum distance of the 3D space is smaller than a first distance threshold value, the target empty parking space is set to be in a non-selectable parking state; and executing a parking action according to the set state of each empty parking space.

Description

Technical Field

[0001] This invention relates to the field of automatic parking technology, and in particular to an automatic parking method and system capable of sensing three-dimensional protruding obstacles. Background Technology

[0002] Automated parking technology is constantly evolving and progressing. It has evolved from the initial ultrasonic radar-based automatic parking assist (APA) to visual spatial parking assist (FAPA), and through continuous technological advancements, it has developed into the mainstream memory parking assist (LAPA), as well as the fully automated valet parking system (AVP) which is under continuous development.

[0003] However, due to current limitations in automatic parking performance, it cannot recognize protruding obstacles in parking lots, such as pipes, fire hydrants, and charging boxes protruding from parking lot walls or pillars. This makes it difficult for existing automatic parking systems, using only ultrasonic radar and cameras without expensive sensor hardware like LiDAR, to accurately detect the precise location of these protruding obstacles around parking spaces. During automatic parking, the vehicle may be scraped by these protruding obstacles, affecting vehicle safety and the user experience of intelligent features. Summary of the Invention

[0004] This invention aims to solve at least one of the aforementioned problems in the prior art by proposing an automatic parking method that integrates 3D protruding obstacle detection. Based on 3D spatial occupancy network technology, it performs high-precision 3D detection of 3D protruding obstacles in parking lots, matches obstacles with target empty parking spaces, determines key parking positions based on the relative positions of obstacles, calculates the 3D physical distance between the protruding obstacle and the outer layer of the vehicle at the key parking position, assesses the risk level, and optimizes and verifies the target parking space position based on the risk level and the physical distance of the 3D protruding obstacle.

[0005] In a first aspect, embodiments of the present invention provide an automatic parking method capable of sensing three-dimensional protruding obstacles, comprising:

[0006] Based on the detection of visual sensors and ultrasonic radar sensors, the fused empty parking spaces, drivable spaces and conventional obstacles are obtained;

[0007] Acquire temporal images from different perspectives of the vehicle's side-view camera, and use a pre-trained 3D panoramic instance segmentation occupancy network model to detect occupancy of 3D protruding obstacles in the parking lot, and obtain 3D panoramic instance segmentation results of the vehicle's surrounding environment including 3D protruding obstacles.

[0008] Based on the vehicle's exterior design, construct dense voxels of the vehicle's surface;

[0009] Iterate through empty parking spaces and protruding obstacles, and add the protruding obstacles that match the empty parking space to the list of matching protruding obstacles for that empty parking space.

[0010] Select the key parking position based on the relative positional relationship between the target three-dimensional protruding obstacle and the matching empty parking space;

[0011] Calculate the minimum 3D spatial distance between the target 3D protruding obstacle and the dense voxel on the side corresponding to the parking key position of the target empty parking space;

[0012] Compare the minimum 3D spatial distance of the target empty parking space with the first distance threshold. When the minimum 3D spatial distance is less than the first distance threshold, set the target empty parking space to be unselectable for parking.

[0013] The parking action is executed based on the status of each available parking space.

[0014] In a preferred embodiment, the method further includes: training a preset 3D panoramic instance segmentation occupancy network model to obtain the pre-trained 3D panoramic instance segmentation occupancy network model.

[0015] In a preferred embodiment, the step of training a pre-set 3D panoramic instance segmentation occupancy network model to obtain the pre-trained 3D panoramic instance segmentation occupancy network model includes:

[0016] Using the PanoOcc occupancy network model, a parking lot dataset including 3D protruding obstacles in the parking lot was collected as the training set, and transfer learning was performed on the pre-set 3D panoramic instance segmentation occupancy network model.

[0017] In the model inference and detection stage, the pre-defined 3D panoramic instance segmentation occupancy network model extracts multi-scale features from multiple viewpoints and frames of images from the camera, obtains voxel features through viewpoint encoders and temporal encoders, and fuses voxel features from different frames into the same scale. Finally, it uses a 3D object detection head and an instance segmentation detection head to detect the 3D features and categories of the target, and uses 3D object detection to optimize object category prediction and assign instance IDs, ultimately obtaining the 3D panoramic instance segmentation occupancy detection results.

[0018] In a preferred embodiment, after the steps of acquiring time-series images from different perspectives of the vehicle side-view camera, using a pre-trained 3D panoramic instance segmentation occupancy network model to detect occupancy of 3D protruding obstacles in the parking lot, and obtaining 3D panoramic instance segmentation results of the vehicle's surrounding environment including 3D protruding obstacles, the method further includes: filtering the 3D panoramic instance segmentation results to retain 3D protruding obstacles.

[0019] The step of filtering the 3D panoramic instance segmentation results to retain 3D protruding obstacles includes:

[0020] The 3D panoramic instance segmentation results are processed and filtered for height, removing those with effective height values ​​exceeding a threshold. The result;

[0021] After high-level processing and filtering, each 3D panoramic instance segmentation result is assessed to determine whether it belongs to a preset type of convex obstacle. If it does, it is listed as a filtered convex obstacle; otherwise, a 3D obstacle convexity assessment is performed: first, the occupied mesh voxels at the ground of the 3D panoramic instance segmentation result are projected onto the ground to obtain the projection area. Then, the overall ground projection is performed on the segmentation result of the 3D panoramic instance to obtain the projection area. Finally, for and Perform area comparison and IOU (Intersection over Union) calculation; when IOU is less than a threshold... and The area is greater than When the area is equal to the specified area, the segmentation result of the 3D panoramic instance is listed as a filtered and selected 3D protruding obstacle.

[0022] In a preferred embodiment, the step of traversing empty parking spaces and protruding obstacles, and adding protruding obstacles that match the empty parking space to the list of matching protruding obstacles for that empty parking space, includes:

[0023] At the four corners of the empty parking space , , The outer perimeter is expanded by a preset range to obtain the temporary outer corner point of the vacant parking space. , , , ;

[0024] Traverse all protruding obstacles and compare their projections on the ground with the temporary outward corner point of the empty parking space. , , , The quadrilateral area is used to determine if there is an intersection. If there is an intersection, the 3D protruding obstacle is added to the list of matching 3D protruding obstacles for the empty parking space.

[0025] In a preferred embodiment, the step of constructing dense voxels of the vehicle surface based on the vehicle's exterior styling includes:

[0026] Using the rear axle center of the vehicle as the reference point , The direction is perpendicular to the car body and to the right. The direction is pointing directly forward of the car. The direction is vertically upward;

[0027] Based on the vehicle's physical dimensions, dense voxels are constructed for the rear, left, right, and top sides of the vehicle surface. Each dense voxel has a volume of 1cm x 1cm x 1cm. The offset of the center point coordinates of each dense voxel relative to the reference origin at the rear axle center is... , , , where i represents different voxel center points and j represents different dense voxels on the outer surface of the vehicle.

[0028] In a preferred embodiment, the step of selecting the key parking location based on the relative positional relationship between the target three-dimensional protruding obstacle and the matching empty parking space includes:

[0029] Generate a virtual 3D bounding box of the vehicle at the target parking position in an empty parking space. The left side is... The right side is Front and rear sides are The top side is The center point of the virtual vehicle's external 3D frame is... The coordinates are , , ;

[0030] Find the six outermost voxels of the target 3D protruding obstacle: top, bottom, left, right, front, back, and back. , , , , ;

[0031] Calculate the center coordinates of the six voxels and the center point of the virtual vehicle's bounding 3D frame. The lines connecting the four faces , , , The intersection points are used to determine the relative position of the three-dimensional protruding obstacle with respect to the matched empty parking space based on the face with the most intersection points. If the number of intersection points is 0, the empty parking space is determined to be unavailable.

[0032] When the target three-dimensional protruding obstacle is located behind or above the matched empty parking space, the key parking position is the target parking position of the vehicle in the empty parking space. When the target three-dimensional obstacle is to the left or right of the matched empty parking space, if the rearview mirror passes through the tangent of the protruding point of the target three-dimensional protruding obstacle during the parking planning process, the key parking position is the position where the rearview mirror passes through the tangent of the protruding point. Otherwise, the key parking position is the target parking position of the vehicle in the empty parking space.

[0033] In a preferred embodiment, the method further includes a preliminary determination of the risk level of vacant parking spaces, wherein the preliminary determination of the risk level of vacant parking spaces includes:

[0034] Set the first distance threshold to be less than the second distance threshold, and set the second distance threshold to be less than the third distance threshold;

[0035] Determine whether there are obstacles in the target empty parking space. If there are no obstacles, the risk level D of the target empty parking space is 0, indicating that there is no risk at present. Otherwise, the risk level D is 1, indicating that there is a slight risk.

[0036] When the risk level D is 1, it is then determined whether there is a matching three-dimensional protruding obstacle in the target empty parking space. If there is, the risk level D of the target empty parking space is updated to 2, indicating that there is a slight risk of protruding obstacle.

[0037] Obtain the minimum 3D spatial distance of all protruding obstacles that match the target empty parking space. ;

[0038] Compare Compared with the third distance threshold, when At that time, the risk level D was updated to 3, indicating a moderate risk of protruding obstacles;

[0039] Compare Compared with the second distance threshold, when At that time, the risk level D was updated to 4, indicating a severe risk of protruding obstacles;

[0040] Compare Compared with the first distance threshold, when At that time, the risk level D is updated to 5, indicating that there is a risk of unparkable protruding obstacles.

[0041] In a preferred embodiment, the step of performing the parking action based on the status of each available parking space includes:

[0042] When the risk level D of the empty parking space to be parked is 3 or 4, the target parking position of the empty parking space is optimized, and the parking action is performed to park the vehicle in the optimized target parking position.

[0043] The optimization of the target parking location for the empty parking space includes:

[0044] The risk levels of the left, right, rear, and top sides of the vehicle are assessed separately to obtain the corresponding risk levels. , , , ;

[0045] First optimize the left and right side positions, if the risk level of the left and right sides is... and If all risk levels are below level 3, no left or right side optimization will be performed. If the risk level on any side reaches level 3, left or right side optimization will be performed. Based on the distance between the side of the vehicle and the edge line of the empty parking space and the maximum allowable side edge overlap, the target parking position will be shifted to the left or right, and the optimized risk level will be updated. , ;

[0046] When the optimized left and right side risk levels and When all are less than 5 and the risk level is on the rear side When the value is greater than or equal to 2, the risk level of the rear side is adjusted according to the new target berthing position. Update the risk level if the updated risk level is... If the risk level is greater than or equal to 3, then the target parking position is optimized both forward and backward. Based on the distance between the edge of the target parking position and the entrance line of the empty parking space and the maximum allowable amount of parking entrance edge overlap, the target parking position is shifted forward, and the optimized risk level is updated. ;

[0047] when When less than 5 and When the risk level is greater than or equal to 2, update the upper risk level. Overall risk level after parking space optimization for: =max( , , , ),when When the value is 5, the target empty parking space will be set to an unselectable parking state.

[0048] In a second aspect, embodiments of the present invention provide an automatic parking system capable of sensing three-dimensional protruding obstacles, the system being able to implement any of the methods described in the first aspect, the system comprising:

[0049] The empty parking space detection module is used to detect empty parking spaces, drivable spaces and conventional obstacles based on visual sensors and ultrasonic radar sensors.

[0050] The 3D protruding obstacle occupancy detection module is used to acquire time-series images from different perspectives of the vehicle side-view camera. It uses a pre-trained 3D panoramic instance segmentation occupancy network model to detect 3D protruding obstacles in the parking lot and obtains 3D panoramic instance segmentation results of the vehicle's surrounding environment, including 3D protruding obstacles.

[0051] The vehicle outer layer dense voxel construction module is used to construct dense voxels of the vehicle surface based on the vehicle's exterior styling.

[0052] The 3D protruding obstacle and empty parking space matching module is used to traverse empty parking spaces and 3D protruding obstacles, and add the 3D protruding obstacles that match the empty parking space to the list of matched 3D protruding obstacles for that empty parking space.

[0053] The parking key position selection module is used to select the parking key position based on the relative positional relationship between the target three-dimensional protruding obstacle and the matching empty parking space.

[0054] The 3D spatial distance calculation module is used to calculate the minimum 3D spatial distance between the target three-dimensional protruding obstacle and the dense voxel on the side corresponding to the parking key position of the target empty parking space.

[0055] The parking space status setting module is used to compare the minimum 3D spatial distance of the target parking space with the first distance threshold. When the minimum 3D spatial distance is less than the first distance threshold, the target parking space is set to be unselectable for parking.

[0056] The parking action module is used to perform parking actions based on the status of each available parking space.

[0057] Thirdly, embodiments of the present invention provide an electronic device, including:

[0058] One or more processors;

[0059] Memory, used to store one or more programs;

[0060] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the first aspect.

[0061] Fourthly, embodiments of the present invention provide a computer-readable medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0062] Beneficial effects of this invention:

[0063] This invention utilizes existing vehicle camera sensors to achieve three-dimensional perception of parking space without increasing hardware costs such as additional sensors. By judging distances in three-dimensional space, it determines the impact of protruding obstacles on the parking space's availability. Combining obstacle information in three-dimensional space, it pre-optimizes available parking spaces and continuously perceives and assesses risks during parking, optimizing the target parking space. This not only increases the application scenarios of automatic parking but also significantly reduces the risk of scratches during parking, improves the safety of automatic parking functions, increases user willingness to use the function and product confidence, and enhances the product image. Attached Figure Description

[0064] Figure 1 A flowchart illustrating an automatic parking method capable of sensing three-dimensional protruding obstacles provided by an embodiment of the present invention. Figure 1 .

[0065] Figure 2 A flowchart illustrating an automatic parking method capable of sensing three-dimensional protruding obstacles provided by an embodiment of the present invention. Figure 2 .

[0066] Figure 3 A flowchart illustrating an automatic parking method capable of sensing three-dimensional protruding obstacles provided by an embodiment of the present invention. Figure 3 .

[0067] Figure 4 This is a structural block diagram of another automatic parking system capable of sensing three-dimensional protruding obstacles, provided as an embodiment of the present invention.

[0068] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0069] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0070] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0071] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0072] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0073] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0074] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0075] In this invention, some technical terms have the following meanings:

[0076] Marked parking spaces: In a parking lot, parking areas are marked with lines using paint or other coatings, or brick-lined parking spaces are formed by dividing different parking spaces with lines made of different colored paving stones.

[0077] Spatial parking space: A parking space constructed in a parking area, regardless of whether there are lines on the ground, based on the area outside the edge of surrounding obstacles or the sufficient gap between obstacles to park a vehicle.

[0078] Three-dimensional protruding obstacles in parking lots: These generally refer to hanging obstacles that often appear near pillars and walls in parking lots, such as water pipes, fire hydrants, fire boxes, charging boxes, charging pillars, wall-mounted charging guns, etc.

[0079] APA: Ultrasonic radar-based automatic parking assist for parking spaces.

[0080] FAPA: Visual Spatial Parking Fusion Automatic Parking Assist.

[0081] LAPA: Automatic Parking Assist with Parking Memory.

[0082] AVP: Valet Parking Assist.

[0083] In related technologies, patent document CN117687041A discloses a method for detecting suspended obstacles in parking lots. This method uses a lidar mounted on the vehicle body to acquire point cloud data of the parking lot environment. The point cloud data is then processed through filtering, plane fitting, clustering, and solving for the minimum bounding cube to obtain the characterization parameters of the minimum bounding cube, thereby enabling the detection of suspended obstacles in the parking lot. The disadvantages of this technical solution are: 1. It uses a lidar sensor, which places high demands on sensor hardware costs and computing power; 2. It only detects protruding obstacles and does not classify the impact of protruding obstacles of different scales on parking; 3. It only detects protruding obstacles and does not consider the non-flatness of the vehicle's exterior surface.

[0084] In related technologies, patent document CN118144810A discloses a vehicle control method, system, and vehicle. The vehicle control method includes: obtaining detection information of the road surface to the side of the vehicle; determining, based on the detection information, whether there are raised or recessed areas on the road surface to the side of the vehicle; if raised or recessed areas exist, obtaining area data of the raised or recessed areas; determining, based on the area data, whether the raised or recessed areas affect the vehicle; if the raised or recessed areas affect the vehicle, providing a warning and / or controlling the vehicle to take evasive action. The disadvantages of this technical solution are: 1. This invention only detects unevenness on conventional road surfaces, but does not detect suspended three-dimensional protruding obstacles around parking spaces; 2. It only considers the impact of ground protruding obstacles on the chassis, etc., and fails to consider the risk of scraping the side of the vehicle body during parking.

[0085] This invention utilizes existing vehicle visual sensors such as surround-view cameras, side-view cameras, and front and rear cameras, as well as ultrasonic radar sensors, without adding high-cost hardware sensors such as lidar. It employs advanced BEV visual occupancy network algorithms (e.g., PanoOcc) to perform 3D panoramic instance segmentation and dense occupancy perception of the parking lot environment. This accurately perceives various common parking obstacles in the parking space, including suspended three-dimensional protruding obstacles. It also focuses on analyzing protruding suspended obstacles near empty parking spaces and calculating the 3D spatial physical distance to key parking positions. This allows for early assessment of parking space risks, optimization of empty space locations, and risk level alerts, further avoiding the risk of collisions. This addresses the common vehicle collision risks caused by blind spots in ultrasonic radar, such as suspended three-dimensional protruding obstacles, during current automatic parking processes.

[0086] Figure 1 This is a flowchart illustrating an automatic parking method capable of sensing three-dimensional protruding obstacles, provided by an embodiment of the present invention; as shown. Figure 1 As shown, the method includes:

[0087] Based on the detection of visual sensors and ultrasonic radar sensors, the fused empty parking spaces, drivable spaces and conventional obstacles are obtained. This detection adopts the traditional method of visual detection of parking space features, drivable space features and conventional obstacle features, combined with ultrasonic detection of space and obstacles. After fusing different features of the same sensor, the detection results of vision and ultrasonic are fused, and finally the fused empty parking spaces, drivable spaces and conventional obstacles are output.

[0088] Acquire temporal images from different perspectives of the vehicle's side-view camera, and use a pre-trained 3D panoramic instance segmentation occupancy network model to detect occupancy of 3D protruding obstacles in the parking lot, and obtain 3D panoramic instance segmentation results of the vehicle's surrounding environment including 3D protruding obstacles.

[0089] Based on the vehicle's exterior design, construct dense voxels of the vehicle's surface;

[0090] Iterate through empty parking spaces and protruding obstacles, and add the protruding obstacles that match the empty parking space to the list of matching protruding obstacles for that empty parking space.

[0091] Select the key parking position based on the relative positional relationship between the target three-dimensional protruding obstacle and the matching empty parking space;

[0092] Calculate the minimum 3D spatial distance between the target 3D protruding obstacle and the dense voxel on the side corresponding to the parking key position of the target empty parking space;

[0093] Compare the minimum 3D spatial distance of the target empty parking space with the first distance threshold. When the minimum 3D spatial distance is less than the first distance threshold, set the target empty parking space to be unselectable for parking.

[0094] The parking action is executed based on the status of each available parking space.

[0095] In some embodiments, such as Figure 2 As shown, it also includes: training a preset 3D panoramic instance segmentation occupancy network model to obtain the pre-trained 3D panoramic instance segmentation occupancy network model.

[0096] In some embodiments, the step of training a preset 3D panoramic instance segmentation occupancy network model to obtain the pre-trained 3D panoramic instance segmentation occupancy network model includes:

[0097] Using the PanoOcc occupancy network model, a parking lot dataset including 3D protruding obstacles in the parking lot was collected as the training set, and transfer learning was performed on the pre-set 3D panoramic instance segmentation occupancy network model.

[0098] In the model inference and detection stage, the pre-defined 3D panoramic instance segmentation occupancy network model extracts multi-scale features from multiple viewpoints and frames of images from the camera, obtains voxel features through viewpoint encoders and temporal encoders, and fuses voxel features from different frames into the same scale. Finally, it uses a 3D object detection head and an instance segmentation detection head to detect the 3D features and categories of the target, and uses 3D object detection to optimize object category prediction and assign instance IDs, ultimately obtaining the 3D panoramic instance segmentation occupancy detection results.

[0099] In some embodiments, such as Figure 3 As shown, the steps of acquiring time-series images from different perspectives of the vehicle side-view camera, using a pre-trained 3D panoramic instance segmentation occupancy network model to detect occupancy of 3D protruding obstacles in the parking lot, and obtaining 3D panoramic instance segmentation results of the vehicle's surrounding environment including 3D protruding obstacles include: filtering the 3D panoramic instance segmentation results to retain 3D protruding obstacles.

[0100] The step of filtering the 3D panoramic instance segmentation results to retain 3D protruding obstacles includes:

[0101] The 3D panoramic instance segmentation results are processed and filtered for height, removing those with effective height values ​​exceeding a threshold. The result;

[0102] After high-level processing and filtering, each 3D panoramic instance segmentation result is assessed to determine whether it belongs to a preset type of convex obstacle. If it does, it is listed as a filtered convex obstacle; otherwise, a 3D obstacle convexity assessment is performed: first, the occupied mesh voxels at the ground of the 3D panoramic instance segmentation result are projected onto the ground to obtain the projection area. Then, the overall ground projection is performed on the segmentation result of the 3D panoramic instance to obtain the projection area. Finally, for and Perform area comparison and IOU (Intersection over Union) calculation; when IOU is less than a threshold... and The area is greater than When the area is equal to the specified area, the segmentation result of the 3D panoramic instance is listed as a filtered and selected 3D protruding obstacle.

[0103] In some embodiments, the step of traversing empty parking spaces and protruding obstacles, and adding protruding obstacles that match the empty parking space to the list of matching protruding obstacles for that empty parking space, includes:

[0104] At the four corners of the empty parking space , , The outer perimeter is expanded by a preset range to obtain the temporary outer corner point of the vacant parking space. , , , ;

[0105] Traverse all protruding obstacles and compare their projections on the ground with the temporary outward corner point of the empty parking space. , , , The quadrilateral area is used to determine if there is an intersection. If there is an intersection, the 3D protruding obstacle is added to the list of matching 3D protruding obstacles for the empty parking space.

[0106] In some embodiments, the step of constructing dense voxels of the vehicle surface based on the vehicle model's exterior styling includes:

[0107] Using the rear axle center of the vehicle as the reference point , The direction is perpendicular to the car body and to the right. The direction is pointing directly forward of the car. The direction is vertically upward;

[0108] Based on the vehicle's physical dimensions, dense voxels are constructed for the rear, left, right, and top sides of the vehicle surface. Each dense voxel has a volume of 1cm x 1cm x 1cm. The offset of the center point coordinates of each dense voxel relative to the reference origin at the rear axle center is... , , , where i represents different voxel center points and j represents different dense voxels on the outer surface of the vehicle.

[0109] In some embodiments, the step of selecting the key parking location based on the relative positional relationship between the target three-dimensional protruding obstacle and the matching empty parking space includes:

[0110] Generate a virtual 3D bounding box of the vehicle at the target parking position in an empty parking space. The left side is... The right side is Front and rear sides are The top side is The center point of the virtual vehicle's external 3D frame is... The coordinates are , , ;

[0111] Find the six outermost voxels of the target 3D protruding obstacle: top, bottom, left, right, front, back, and back. , , , , ;

[0112] Calculate the center coordinates of the six voxels and the center point of the virtual vehicle's bounding 3D frame. The lines connecting the four faces , , , The intersection points are used to determine the relative position of the three-dimensional protruding obstacle with respect to the matched empty parking space based on the face with the most intersection points. If the number of intersection points is 0, the empty parking space is determined to be unavailable.

[0113] When the target three-dimensional protruding obstacle is located behind or above the matched empty parking space, the key parking position is the target parking position of the vehicle in the empty parking space. When the target three-dimensional obstacle is to the left or right of the matched empty parking space, if the rearview mirror passes through the tangent of the protruding point of the target three-dimensional protruding obstacle during the parking planning process, the key parking position is the position where the rearview mirror passes through the tangent of the protruding point. Otherwise, the key parking position is the target parking position of the vehicle in the empty parking space.

[0114] In some embodiments, the method further includes a preliminary determination of the risk level of vacant parking spaces, wherein the preliminary determination of the risk level of vacant parking spaces includes:

[0115] Set the first distance threshold to be less than the second distance threshold, and set the second distance threshold to be less than the third distance threshold;

[0116] Determine whether there are obstacles in the target empty parking space. If there are no obstacles, the risk level D of the target empty parking space is 0, indicating that there is no risk at present. Otherwise, the risk level D is 1, indicating that there is a slight risk.

[0117] When the risk level D is 1, it is then determined whether there is a matching three-dimensional protruding obstacle in the target empty parking space. If there is, the risk level D of the target empty parking space is updated to 2, indicating that there is a slight risk of protruding obstacle.

[0118] Obtain the minimum 3D spatial distance of all protruding obstacles that match the target empty parking space. ;

[0119] Compare Compared with the third distance threshold, when At that time, the risk level D was updated to 3, indicating a moderate risk of protruding obstacles;

[0120] Compare Compared with the second distance threshold, when At that time, the risk level D was updated to 4, indicating a severe risk of protruding obstacles;

[0121] Compare Compared with the first distance threshold, when At that time, the risk level D is updated to 5, indicating that there is a risk of unparkable protruding obstacles.

[0122] In some embodiments, the step of performing a parking action based on the status of each available parking space includes:

[0123] When the risk level D of the empty parking space to be parked is 3 or 4, the target parking position of the empty parking space is optimized, and the parking action is performed to park the vehicle in the optimized target parking position.

[0124] The optimization of the target parking location for the empty parking space includes:

[0125] The risk levels of the left, right, rear, and top sides of the vehicle are assessed separately to obtain the corresponding risk levels. , , , ;

[0126] First optimize the left and right side positions, if the risk level of the left and right sides is... and If all risk levels are below level 3, no left or right side optimization will be performed. If the risk level on any side reaches level 3, left or right side optimization will be performed. Based on the distance between the side of the vehicle and the edge line of the empty parking space and the maximum allowable side edge overlap, the target parking position will be shifted to the left or right, and the optimized risk level will be updated. , ;

[0127] When the optimized left and right side risk levels and When all are less than 5 and the risk level is on the rear side When the value is greater than or equal to 2, the risk level of the rear side is adjusted according to the new target berthing position. Update the risk level if the updated risk level is... If the risk level is greater than or equal to 3, then the target parking position is optimized both forward and backward. Based on the distance between the edge of the target parking position and the entrance line of the empty parking space and the maximum allowable amount of parking entrance edge overlap, the target parking position is shifted forward, and the optimized risk level is updated. ;

[0128] when When less than 5 and When the risk level is greater than or equal to 2, update the upper risk level. Overall risk level after parking space optimization for: =max( , , , ),when When the value is 5, the target empty parking space will be set to an unselectable parking state.

[0129] Based on the same inventive concept, embodiments of the present invention also provide an automatic parking system capable of sensing three-dimensional protruding obstacles. The system is capable of implementing any of the methods described in the above embodiments, and includes:

[0130] The empty parking space detection module is used to detect empty parking spaces, drivable spaces and conventional obstacles based on visual sensors and ultrasonic radar sensors.

[0131] The 3D protruding obstacle occupancy detection module is used to acquire time-series images from different perspectives of the vehicle side-view camera. It uses a pre-trained 3D panoramic instance segmentation occupancy network model to detect 3D protruding obstacles in the parking lot and obtains 3D panoramic instance segmentation results of the vehicle's surrounding environment, including 3D protruding obstacles.

[0132] The vehicle outer layer dense voxel construction module is used to construct dense voxels of the vehicle surface based on the vehicle's exterior styling.

[0133] The 3D protruding obstacle and empty parking space matching module is used to traverse empty parking spaces and 3D protruding obstacles, and add the 3D protruding obstacles that match the empty parking space to the list of matched 3D protruding obstacles for that empty parking space.

[0134] The parking key position selection module is used to select the parking key position based on the relative positional relationship between the target three-dimensional protruding obstacle and the matching empty parking space.

[0135] The 3D spatial distance calculation module is used to calculate the minimum 3D spatial distance between the target three-dimensional protruding obstacle and the dense voxel on the side corresponding to the parking key position of the target empty parking space.

[0136] The parking space status setting module is used to compare the minimum 3D spatial distance of the target parking space with the first distance threshold. When the minimum 3D spatial distance is less than the first distance threshold, the target parking space is set to be unselectable for parking.

[0137] The parking action module is used to perform parking actions based on the status of each available parking space.

[0138] Based on the same inventive concept, embodiments of the present invention also provide, such as Figure 4 Another type of automated parking system shown is capable of sensing three-dimensional protruding obstacles. This system includes: an empty parking space detection module, a 3D obstacle occupancy detection module, a protruding obstacle filtering module, a protruding obstacle and empty parking space matching module, a vehicle outer layer dense voxel construction module, a parking key location selection module, a calculated feature point filtering module, a 3D spatial distance calculation module, a risk level initial determination module, and a parking space pre-optimization and risk update module.

[0139] The empty parking space detection module adopts the traditional method of visually detecting parking space features, drivable space, and conventional obstacle features, combined with ultrasonic detection of space and obstacles. It fuses different features from the same sensor, and then fuses the detection results of vision and ultrasound, finally outputting the fused empty parking space, drivable space, and conventional obstacles.

[0140] The 3D obstacle occupancy detection module uses temporal images from different perspectives of a vehicle side-view camera as input and employs a 3D panoramic instance segmentation occupancy network model to detect 3D obstacles in parking lots. For example, using the PanoOcc occupancy network model, a parking lot dataset including protruding obstacles is collected as the training set. Transfer learning is performed on the 3D panoramic instance segmentation occupancy network model to make it more suitable for parking lot scenarios. In the model inference and detection stage, the model extracts multi-scale features from multi-view and multi-frame images from the camera, obtains voxel features through a view encoder and a temporal encoder, and fuses voxel features from different frames to the same scale. Finally, a 3D object detection head and an instance segmentation detection head are used to detect the 3D features and categories of the target. 3D object detection is used to optimize object category prediction and assign instance IDs, ultimately obtaining the 3D panoramic instance segmentation occupancy detection result. The position coordinates of a single 3D protruding obstacle target are represented as [( , , ), ( , , ), ...], where i represents the instance ID. For the 3D obstacle occupancy detection module in this invention, the output result is the three-dimensional panoramic instance segmentation result of the vehicle's surrounding environment, including protruding obstacles in the parking lot.

[0141] The protruding obstacle filtering module filters the 3D panoramic instance segmentation results to retain 3D protruding obstacles. Firstly, to reduce computational load, the 3D panoramic instance segmentation results are height-processed and filtered, removing valid height values ​​exceeding a threshold. The result, threshold The selection can be based on parking lot design specifications, for example, setting an effective height threshold of 220cm. After height filtering, each 3D instance segmentation result is processed separately. First, a category judgment is performed to determine whether it belongs to a common protruding obstacle type (such as fire pipes, fire boxes, charging boxes, charging guns, etc.). If it does, it is listed as a filtered protruding obstacle. If it is not a common protruding obstacle type, a 3D obstacle protrusion judgment is performed. First, the ground-occupied mesh voxels at the obstacle's grounding point are projected to obtain the projection area. Then, a global ground projection is performed on the 3D obstacle instance to obtain the projection area. Finally, for and Perform area comparison and IOU (Intersection over Union), calculate the intersection-over-union ratio, and when the IOU is less than a threshold... and area Greater than area At that time, the 3D obstacle after filtering the height is listed as the filtered protruding obstacle, and the category attribute is updated to other protruding obstacles.

[0142] The module for matching protruding obstacles with available parking spaces cycles through available parking spaces, matching the four corner points of each available parking space. , , Expand the outer area to a certain extent, specifically by using... For example, if it is necessary to expand outwards... Distance, first calculate the coordinates of the center point of the parking space. Then make a pass A straight line from a point Solve for the distance to the line. Distance is Two parallel lines and Calculate separately And two parallel lines and distance and Parallel lines that are further apart are straight lines. Outward expansion straight line Similarly, based on the other three straight lines , , Corresponding outward expansion distance , , The requirement is to find the corresponding outward expansion line. , , The intersection of the four outward-expanding straight lines is the temporary outward-expanding corner point of the empty parking space. , , , ), iterate through the filtered protruding obstacles, and compare the projection of the protruding obstacle on the ground with the temporary outer corner point of the empty parking space ( , , , The system performs intersection checks on the quadrilateral area enclosed by the parking space. If an intersection exists, the protruding obstacle is added to the list of matching protruding obstacles for that empty parking space. The system also determines the position of the protruding obstacle relative to the parking space, such as front, rear, left, right, or top. One protruding obstacle can correspond to multiple empty parking spaces, and one empty parking space can also correspond to multiple protruding obstacles.

[0143] Based on the vehicle's exterior shape, the dense voxel construction module constructs dense voxels for the rear, left, right, and top sides of the vehicle surface (the front and bottom sides are less affected by protruding obstacles and can be constructed as needed). The specific construction method is as follows: using the center of the vehicle's rear axle as the reference origin. , The direction is perpendicular to the car body and to the right. The direction is pointing directly forward of the car. The direction is vertically upward, and dense voxels are constructed on each side surface of the vehicle with reference to the actual physical dimensions of the car model. The individual volume of each dense voxel is 1cm*1cm*1cm. The offset of the center point coordinates of each dense voxel relative to the reference origin of the rear axle center of the vehicle is ( , , ), where i represents different voxel center points, and j represents different dense voxels on the outer surface of the vehicle. Taking the right side of the vehicle as an example, the relative coordinates of the dense voxels on the right side of the vehicle are [( , , ), ( , , ),......].

[0144] The parking key position selection module first determines the relative position of the target obstacle with respect to the matched empty parking space. Based on the relative positional relationship between the protruding obstacle and the empty parking space, it selects the key parking position and performs subsequent distance calculations at the key parking position. The method for determining the relative positional relationship between the protruding obstacle and the target empty parking space is as follows: a virtual 3D bounding box of the vehicle is generated at the target parking position in the empty parking space, with the left side as... The right side is Front and rear sides are The top side is The center point of the virtual vehicle's external 3D frame is... ( , , For a target protruding from an obstacle, solve for the six outermost voxels of the target protruding from the obstacle (top, bottom, left, right, front, back, and back). , , , , ), calculate the center coordinates of the 6 voxels and the center point of the virtual vehicle's circumscribed 3D frame respectively. The lines connecting the four faces ( , , , The parking space is determined by the intersection of the faces with the most intersections. If all faces have zero intersections, the parking space is deemed unusable and eliminated. In the special case of multiple faces having the same intersection, the protruding obstacle is considered to have multiple relative positions. For selecting the key parking position, taking a perpendicular parking space as an example, if the target protruding obstacle is located behind or above the matched parking space, the key parking position is the vehicle's final parking target position. If the target obstacle is to the left or right of the matched parking space, the key parking position must consider the rearview mirror position. If the rearview mirror passes through the tangent of the protruding obstacle's protrusion point during the parking planning process, the key parking position is the position where the rearview mirror passes through the tangent of the protrusion point; otherwise, the key parking position is the vehicle's final parking target position.

[0145] To reduce the computational load of calculating the minimum 3D physical distance, this invention proposes a spatial distance calculation optimization method. Specifically, based on positioning information such as odometer readings, a coordinate system with the rear axle center as the origin is obtained for key parking positions. , , The positional relationship of the target 3D protruding obstacle relative to the coordinate system of its spatial location is transformed into a coordinate system with the axis center as the origin. , , If the target protrudes from the obstacle, then at this time, the position coordinates of the 3D target protruding from the obstacle target are [( , , ), ( , , ),......] convert to [( , , ), ( , , ), ...], taking the obstacle protruding on the right side of the vehicle as an example, its corresponding coordinate system ( , , )of The direction is the position coordinate of the 3D protruding obstacle target [( , , ), ( , , Sort the x values ​​in […] and take the median value. With median value To define the boundary, only the coordinates of the target's 3D protrusion from the obstacle are retained in the coordinate system closest to it. , , Half of the origin; at the same time, due to the coordinates of the protruding obstacle on the right and in The plane and the right side of the vehicle theoretically overlap, so the position coordinates of the target's 3D protrusion from the obstacle are calculated [( , , ), ( , , The range of values ​​for (y, z) in […] ]and[ ], then retain the relative coordinates of the dense voxels on the right side surface of the vehicle [( , , ), ( , , The range of values ​​for y in [ ), ...] is [ And the range of values ​​for z is [ The points are defined by β, where β is the expansion threshold. Similarly, the range of (y,z) is calculated for the left convex obstacle, the range of (x,z) is calculated for the rear convex obstacle, and the range of (x,y) is calculated for the upper convex obstacle, and then subsequent calculations are performed.

[0146] The 3D spatial distance calculation module calculates the minimum 3D spatial distance between the target protruding obstacle and the corresponding dense voxel 3D space of the key parking position of the target empty parking space. Specifically, taking a 3D protruding obstacle located on the right side of the vehicle as an example, the filtered 3D protruding obstacle points [( , , ), ( , , ),......] and the relative coordinates of the dense voxels on the right side surface of the filtered vehicle [( , , ), ( , , Calculate the minimum 3D physical distance between ), ... Where i represents the protruding obstacle ID, and r represents the right side. If there are multiple 3D protruding obstacles to the right of the critical parking position, the minimum 3D physical distance to each protruding obstacle relative to the right side is calculated. , ,......), where (i, j...) represents the protruding obstacle ID, then the minimum 3D physical distance corresponding to the right side of the key parking position. The calculation formula is as follows, where m represents the protruding obstacle ID corresponding to the minimum 3D physical distance.

[0147] =min( , , ......)

[0148] Similarly, the minimum 3D physical distances to the left, rear, and top of key parking positions can be calculated using the following formulas:

[0149] =min( , , ......)

[0150] =min( , , ......)

[0151] =min( , , ......)

[0152] The minimum 3D physical distance of all 3D protruding obstacles matching the overall target empty parking space. The calculations are as follows, with the default value for each minimum 3D physical distance being min(α, ), where α is the default distance value when there are no obstacles. The minimum 2D planar spatial distance between each side of a common non-protruding obstacle, calculated using conventional methods.

[0153] =min( , , , )

[0154] The initial risk level assessment module first sets three risk distance thresholds ( , , ),in Less than , Less than Its value can be set with reference to the accuracy of 3D obstacle detection and parking positioning control. For example, it can be set to... It is 3cm. It is 10cm. The distance is 30cm. The initial risk level determination steps are as follows: First, determine whether there are obstacles around the target parking space. If not, the risk level of the empty parking space is 0, indicating no risk at present; otherwise, the risk level D is 1, indicating a slight risk. Second, when the risk level D is 1, determine whether there are matching 3D protruding obstacles around the risky vehicle. If so, the risk level D of the target parking space is updated to 2, indicating a slight risk from protruding obstacles. Third, compare... and ,when At that time, the risk level D is updated to 3, indicating a moderate risk of protruding obstacles; the fourth step is to compare. and ,when At that time, the risk level D was updated to 4, indicating a severe risk of protruding obstacles; the fifth step was to compare... and ,when At that time, the risk level D is updated to 5, indicating that there is a risk of unparkable protruding obstacles.

[0155] The parking space pre-optimization and risk update module optimizes the target parking space for empty parking spaces when the risk level D≥3. This is because the minimum distance to the 3D obstacle corresponding to the target parking position of the target empty parking space is known. , , , (For reference only) The rules for initial risk level determination involve separately assessing the risk level of each side of the vehicle's risk to obtain the corresponding risk level. , , , The optimization method is to first optimize the left and right sides, and then, if the risk level of the left and right sides is low... and If all risk levels are below 3, no left or right side optimization will be performed. If the risk level on either side reaches 3, then left or right side optimization will be performed. For example, if the risk level on the left side is... For vehicles at or above level 3, calculate the distance between the right side of the vehicle and the right line of the empty parking space. Then the minimum distance to the 3D obstacle on the right. Updated to Then calculate the sum of the distances on the left and right sides. ,but and The calculation formula is as follows, where γ is the maximum allowable amount of parking side line pressure by the system.

[0156] =min( +γ, )

[0157] =( + )

[0158] Subsequently, With 2 2 2 The size relationship. When ≥2 At that time, the target parking position will be shifted to the right. - The distance is [distance], at which point the risk level on the left is [value]. Updated to level 2, risk level on the right. Unchanged (2 or less); when ∈[2 ,2 When the target is moored, the target position will be shifted to the right. / 2- The distance (shifted in the opposite direction when negative) determines the risk level on the left and right sides. and Updated to 3, when ∈[ ,2 When the target is moored, the target position will be shifted to the right. / 2- The distance between the two sides determines the risk level on both sides. and Updated to 4. ≤2 Move the target parking position to the right ( / 2- The distance between the two sides determines the risk level on both sides. and The risk level has been updated to 5, indicating a non-parking obstacle protrusion. This applies to the optimized left and right side risk levels. and When all are <5 and the risk level is 4, the risk level is 4. When the value is ≥2, indicating the presence of a 3D protruding obstacle at the rear, the rear risk is checked. Based on the new target parking position, algorithms from the feature point filtering module and the 3D spatial distance calculation module are used to calculate the minimum distance to the new rear obstacle. Then, based on the initial risk level assessment module, the risk level is further assessed. Update the risk level if the updated risk level is... If the value is ≥3, then the target parking position will be optimized forward and backward. The default distance between the edge of the target parking space and the entrance line of the empty parking space is . The system is set to allow a maximum allowable amount of line pressure at the parking entrance edge, δ, thus achieving the maximum adjusted backward distance of [value missing]. The calculation formula is as follows:

[0159] = + +δ

[0160] Compare and The size, when When the target parking space is moved forward and horizontally ( - Distance, at this time the risk level on the rear. Updated to 2; when ∈[ , When the target parking space is moved forward and then horizontally ( ), the target parking space will be moved forward and horizontally ( ). - Distance, at this time the risk level on the rear. Updated to 3; when ∈[ , When the target parking space is moved forward and then horizontally ( ), the target parking space will be moved forward and horizontally ( ). - Distance, at this time the risk level on the rear. Updated to 4; when When the target parking space is moved forward and horizontally ( - Distance, at this time the risk level on the rear. Updated to 5. (When the backward risk level...) <5 and the upper risk level When the value is ≥2, indicating the presence of a 3D protruding obstacle on the upper side, the upper side risk is checked. Based on the new target parking position, algorithms from the feature point filtering module and the 3D spatial distance calculation module are used to calculate the minimum distance to the new upper obstacle. Then, based on the initial risk level assessment module, the upper risk level is determined and updated accordingly. Overall risk level after parking space optimization as follows:

[0161] =max( , , , )

[0162] Automatic parking systems can adjust according to the current situation. Different values ​​will trigger risk warnings for users. When the value is 5, for safety reasons, the target empty parking space needs to be set to an unselectable parking state. When the parking system is in a safer mode, it needs to be set to an unselectable parking state. When the value is 4, the target empty parking space will also be set to an unselectable parking state.

[0163] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 5 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0164] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0165] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0166] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0167] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable medium. This computer-readable medium stores a computer program, wherein, when executed by a processor, the program implements the steps of any of the methods described in the above embodiments. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0168] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0169] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0170] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0171] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0172] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0173] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0174] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0175] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0176] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0177] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. An automatic parking method capable of sensing three-dimensional protruding obstacles, characterized in that, include: Based on the detection of visual sensors and ultrasonic radar sensors, the fused empty parking spaces, drivable spaces and conventional obstacles are obtained; Acquire temporal images from different perspectives of the vehicle's side-view camera, and use a pre-trained 3D panoramic instance segmentation occupancy network model to detect occupancy of 3D protruding obstacles in the parking lot, and obtain 3D panoramic instance segmentation results of the vehicle's surrounding environment including 3D protruding obstacles. Based on the vehicle's exterior design, construct dense voxels of the vehicle's surface; Iterate through empty parking spaces and protruding obstacles, and add the protruding obstacles that match the empty parking space to the list of matching protruding obstacles for that empty parking space. Select the key parking position based on the relative positional relationship between the target three-dimensional protruding obstacle and the matching empty parking space; Calculate the minimum 3D spatial distance between the target 3D protruding obstacle and the dense voxel on the side corresponding to the parking key position of the target empty parking space; Compare the minimum 3D spatial distance of the target empty parking space with the first distance threshold. When the minimum 3D spatial distance is less than the first distance threshold, set the target empty parking space to be unselectable for parking. The parking action is executed based on the status of each available parking space.

2. The method according to claim 1, characterized in that, Also includes: The pre-trained 3D panoramic instance segmentation occupancy network model is obtained by training the preset 3D panoramic instance segmentation occupancy network model.

3. The method according to claim 2, characterized in that, The steps for training a pre-defined 3D panoramic instance segmentation occupancy network model to obtain the pre-trained 3D panoramic instance segmentation occupancy network model include: Using the PanoOcc occupancy network model, a parking lot dataset including 3D protruding obstacles in the parking lot was collected as the training set, and transfer learning was performed on the pre-set 3D panoramic instance segmentation occupancy network model. In the model inference and detection stage, the pre-defined 3D panoramic instance segmentation occupancy network model extracts multi-scale features from multiple viewpoints and frames of images from the camera, obtains voxel features through viewpoint encoders and temporal encoders, and fuses voxel features from different frames into the same scale. Finally, it uses a 3D object detection head and an instance segmentation detection head to detect the 3D features and categories of the target, and uses 3D object detection to optimize object category prediction and assign instance IDs, ultimately obtaining the 3D panoramic instance segmentation occupancy detection results.

4. The method according to claim 1, characterized in that, The steps of acquiring time-series images from different perspectives of the vehicle side-view camera, using a pre-trained 3D panoramic instance segmentation occupancy network model to detect occupancy of 3D protruding obstacles in the parking lot, and obtaining 3D panoramic instance segmentation results of the vehicle's surrounding environment including 3D protruding obstacles include: filtering the 3D panoramic instance segmentation results to retain 3D protruding obstacles. The step of filtering the 3D panoramic instance segmentation results to retain 3D protruding obstacles includes: The 3D panoramic instance segmentation results are processed and filtered for height, removing those with effective height values ​​exceeding a threshold. The result; After high-level processing and filtering, each 3D panoramic instance segmentation result is assessed to determine whether it belongs to a preset type of convex obstacle. If it does, it is listed as a filtered convex obstacle; otherwise, a 3D obstacle convexity assessment is performed: first, the occupied mesh voxels at the ground of the 3D panoramic instance segmentation result are projected onto the ground to obtain the projection area. Then, the overall ground projection is performed on the segmentation result of the 3D panoramic instance to obtain the projection area. Finally, for and Perform area comparison and IOU (Intersection over Union) calculation; when IOU is less than a threshold... and The area is greater than When the area is equal to the specified area, the segmentation result of the 3D panoramic instance is listed as a filtered and selected 3D protruding obstacle.

5. The method according to claim 1, characterized in that, The step of traversing empty parking spaces and protruding obstacles, and adding the protruding obstacles that match the empty parking space to the list of matching protruding obstacles for that empty parking space includes: At the four corners of the empty parking space , , The outer perimeter is expanded by a preset range to obtain the temporary outer corner point of the vacant parking space. , , , ; Traverse all protruding obstacles and compare their projections on the ground with the temporary outward corner point of the empty parking space. , , , The quadrilateral area is used to determine if there is an intersection. If there is an intersection, the 3D protruding obstacle is added to the list of matching 3D protruding obstacles for the empty parking space.

6. The method according to claim 1, characterized in that, The step of constructing dense voxels of the vehicle surface based on the vehicle's exterior styling includes: Using the rear axle center of the vehicle as the reference point , The direction is perpendicular to the car body and to the right. The direction is pointing directly forward of the car. The direction is vertically upward; Based on the vehicle's physical dimensions, dense voxels are constructed for the rear, left, right, and top sides of the vehicle surface. Each dense voxel has a volume of 1cm x 1cm x 1cm. The offset of the center point coordinates of each dense voxel relative to the reference origin at the rear axle center is... , , , where i represents different voxel center points and j represents different dense voxels on the outer surface of the vehicle.

7. The method according to claim 1, characterized in that, The step of selecting the key parking position based on the relative positional relationship between the target three-dimensional protruding obstacle and the matching empty parking space includes: Generate a virtual 3D bounding box of the vehicle at the target parking position in an empty parking space. The left side is... The right side is Front and rear sides are The top side is The center point of the virtual vehicle's external 3D frame is... The coordinates are , , ; Find the six outermost voxels of the target 3D protruding obstacle: top, bottom, left, right, front, back, and back. , , , , ; Calculate the center coordinates of the six voxels and the center point of the virtual vehicle's bounding 3D frame. The lines connecting the four faces , , , The intersection points are used to determine the relative position of the three-dimensional protruding obstacle with respect to the matched empty parking space based on the face with the most intersection points. If the number of intersection points is 0, the empty parking space is determined to be unavailable. When the target three-dimensional protruding obstacle is located behind or above the matched empty parking space, the key parking position is the target parking position of the vehicle in the empty parking space. When the target three-dimensional obstacle is to the left or right of the matched empty parking space, if the rearview mirror passes through the tangent of the protruding point of the target three-dimensional protruding obstacle during the parking planning process, the key parking position is the position where the rearview mirror passes through the tangent of the protruding point. Otherwise, the key parking position is the target parking position of the vehicle in the empty parking space.

8. The method according to claim 1, characterized in that, It also includes a preliminary assessment of the risk level of vacant parking spaces, the steps of which include: Set the first distance threshold to be less than the second distance threshold, and set the second distance threshold to be less than the third distance threshold; Determine whether there are obstacles in the target empty parking space. If there are no obstacles, the risk level D of the target empty parking space is 0, indicating that there is no risk at present. Otherwise, the risk level D is 1, indicating that there is a slight risk. When the risk level D is 1, it is then determined whether there is a matching three-dimensional protruding obstacle in the target empty parking space. If there is, the risk level D of the target empty parking space is updated to 2, indicating that there is a slight risk of protruding obstacle. Obtain the minimum 3D spatial distance of all protruding obstacles that match the target empty parking space. ; Compare Compared with the third distance threshold, when At that time, the risk level D was updated to 3, indicating a moderate risk of protruding obstacles; Compare Compared with the second distance threshold, when At that time, the risk level D was updated to 4, indicating a severe risk of protruding obstacles; Compare Compared with the first distance threshold, when At that time, the risk level D is updated to 5, indicating that there is a risk of unparkable protruding obstacles.

9. The method according to claim 8, characterized in that, The steps for performing parking actions based on the status of each available parking space include: When the risk level D of the empty parking space to be parked is 3 or 4, the target parking position of the empty parking space is optimized, and the parking action is performed to park the vehicle in the optimized target parking position. The optimization of the target parking location for the empty parking space includes: The risk levels of the left, right, rear, and top sides of the vehicle are assessed separately to obtain the corresponding risk levels. , , , ; First optimize the left and right side positions, if the risk level of the left and right sides is... and If all risk levels are below level 3, no left or right side optimization will be performed. If the risk level on any side reaches level 3, left or right side optimization will be performed. Based on the distance between the side of the vehicle and the edge line of the empty parking space and the maximum allowable side edge overlap, the target parking position will be shifted to the left or right, and the optimized risk level will be updated. , ; When the optimized left and right side risk levels and When all are less than 5 and the risk level is on the rear side When the value is greater than or equal to 2, the risk level of the rear side is adjusted according to the new target berthing position. Update the risk level if the updated risk level is... If the risk level is greater than or equal to 3, then the target parking position is optimized both forward and backward. Based on the distance between the edge of the target parking position and the entrance line of the empty parking space and the maximum allowable amount of parking entrance edge overlap, the target parking position is shifted forward, and the optimized risk level is updated. ; when When less than 5 and When the risk level is greater than or equal to 2, update the upper risk level. Overall risk level after parking space optimization for: =max( , , , ),when When the value is 5, the target empty parking space will be set to an unselectable parking state.

10. An automatic parking system capable of sensing three-dimensional protruding obstacles, characterized in that, The system is capable of implementing the method as described in any one of claims 1 to 9, and the system comprises: The empty parking space detection module is used to detect empty parking spaces, drivable spaces and conventional obstacles based on visual sensors and ultrasonic radar sensors. The 3D protruding obstacle occupancy detection module is used to acquire time-series images from different perspectives of the vehicle side-view camera. It uses a pre-trained 3D panoramic instance segmentation occupancy network model to detect 3D protruding obstacles in the parking lot and obtains 3D panoramic instance segmentation results of the vehicle's surrounding environment, including 3D protruding obstacles. The vehicle outer layer dense voxel construction module is used to construct dense voxels of the vehicle surface based on the vehicle's exterior styling. The 3D protruding obstacle and empty parking space matching module is used to traverse empty parking spaces and 3D protruding obstacles, and add the 3D protruding obstacles that match the empty parking space to the list of matched 3D protruding obstacles for that empty parking space. The parking key position selection module is used to select the parking key position based on the relative positional relationship between the target three-dimensional protruding obstacle and the matching empty parking space. The 3D spatial distance calculation module is used to calculate the minimum 3D spatial distance between the target three-dimensional protruding obstacle and the dense voxel on the side corresponding to the parking key position of the target empty parking space. The parking space status setting module is used to compare the minimum 3D spatial distance of the target parking space with the first distance threshold. When the minimum 3D spatial distance is less than the first distance threshold, the target parking space is set to be unselectable for parking. The parking action module is used to perform parking actions based on the status of each available parking space.

11. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 9.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 9.

Images