A parking space intelligent detection method based on open environment open set instance perception

Abstract

The application provides a parking space intelligent detection method based on open environment open set instance perception, fully perceives ground and above-ground environment information based on front view images and bird's-eye view images; open vocabulary detection and segmentation technology is adopted, and excellent open set perception capability is achieved in an open environment; instance post-processing is carried out on a parking line mask, parking line instances are represented in the form of line segments, and subsequent instantiation reasoning is facilitated; combined with laser data, a three-dimensional global open set instance representation of the environment is established; combined with language description provided by a user, a large language model identifies parking space judgment objects and judgment conditions, and has the ability to judge parking spaces according to language interaction; object instances meeting the conditions are retrieved from a global map, and parking spaces are inferred through a geometric method according to object relationships, so that the optimality and rationality of parking space detection are guaranteed. The method finally improves the parking space recognition capability in an open environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of automobile parking space recognition, and in particular relates to an intelligent parking space detection method based on open environment open set instance perception. Background Art

[0002] Parking technologies such as automated valet parking and memory parking have gained widespread popularity due to their improved user experience and parking efficiency, and are widely used in standard parking lots with clearly marked parking spaces. Parking space detection, as a key component of this task, plays a crucial role in completing parking tasks. However, real-world parking environments are complex and open, such as those in neighborhoods and on the street with unmarked or visible parking lines. This can negatively impact AVP (Automated Valet Parking) systems, particularly in terms of parking space detection.

[0003] Currently, parking space detection can be roughly divided into two types: free space-based methods and visual marker-based methods. Free space-based methods are mainly based on vehicle posture or local geometric free space, and generally use sensors such as lidar, binocular cameras, and ultrasonic waves to perceive environmental geometric information. These methods are difficult to operate in environments without adjacent vehicles or large open areas, and do not perform semantic understanding of parking lines. Visual marker-based methods rely on visual detection of parking space markings, usually using traditional technologies to detect parking line corners and segments, and then analyze the complete parking space, or directly use deep learning-based methods to directly identify parking spaces. Some methods also provide information such as parking space occupancy and entry points, but these methods usually assume that parking line markings exist and are clearly visible, and are prone to detection failure when parking lines are unclear, damaged, or non-existent.

[0004] In the aforementioned problematic scenarios, although existing parking space detection methods perform poorly or even fail, people can easily identify valid parking spaces. Specifically, parking space inference should combine subjective definitions with common sense. For example, even if the parking lines are damaged or there are no visible parking line markings, the empty space between two parallel vehicles or next to a wall or roadside is likely to be considered a valid parking space. In short, people will park their cars in any parking location that meets the common sense definition of parking. These common sense definitions vary and are often based on semantic descriptions and object-behavior relationships. For example, a rectangular empty space on the roadside can be used as a parking space. Therefore, in open scenes, identifying these various complex parking spaces requires a thorough understanding of the environment, combined with inference based on the behavioral relationships between objects. This poses a significant challenge to current parking space detection methods. Summary of the Invention

[0005] In light of this, the present invention provides an intelligent parking space detection method based on open-set instance perception in an open environment. This method uses open-set instance perception of the environment from front-view and bird's-eye view image data, analyzes instance-behavior relationships using a large language model, and flexibly infers various parking spaces. This method is adaptable to different parking space judgment criteria and is applicable to both standard parking lots and non-standard parking environments where parking space lines are blurred, damaged, or non-existent.

[0006] An intelligent parking space detection method based on open environment open set instance perception includes:

[0007] Step 1) Obtain the environmental lidar point cloud and forward-looking camera information. For the forward-looking camera image, obtain the corresponding bird's-eye view image through inverse perspective mapping;

[0008] Step 2) Using open vocabulary detection and segmentation technology on the front view image and the bird's-eye view image, different prompt word technologies are used to identify instance masks related to parking spaces, such as vehicles and parking lines. The parking line masks are then instantiated and post-processed to obtain a complete image instance description.

[0009] Step 3) combining the open set instance perception results of the front view image and the bird's-eye view image obtained in step 2) with the laser point cloud data to construct a three-dimensional global instance map including the ground and above-ground objects;

[0010] In step 4, the large language model identifies the parking space judgment object type and parking space judgment criteria including distance and direction based on the parking space judgment conditions provided by the user. It then searches for object instances that meet the conditions in the global instance map obtained in step 3) and uses geometric reasoning based on object relationships to obtain parking spaces. The final parking space is selected based on the priority of parking space types and common sense knowledge about parking space existence.

[0011] Preferably, the step 2) specifically includes:

[0012] Step 2.1) Instance object detection. Use the object category hint as input to the Grounding DINO model for ground object detection in the front view image. Use the predefined text description hint as input to the Grounding DINO model for ground object detection in the bird's-eye view image. Finally, obtain the object detection bounding boxes and confidence scores for the front view and bird's-eye view images of the current frame.

[0013] Step 2.2) Instance Object Mask Segmentation: For the detection results of the front view image and the bird's-eye view image obtained in step 2.1), use the TAP model to obtain the image mask and title features corresponding to each detection result in the front view image and the bird's-eye view image of the current frame;

[0014] Step 2.3) Post-processing of the stop line mask: Based on the current frame instance object mask obtained in step 2.2), the stop line mask is retrieved from the mask's title features, and the outer contour corresponding to the mask is obtained; each stop line mask outer contour is equidistantly sampled to obtain a set of equidistant sampling point positions. For each equidistant sampling point, the angle between the two adjacent sampling points is used as the angle value of the sampling point. The angle is limited to 0° to 180°, and the angle set corresponding to the equidistant sampling points is obtained. Among them, p i represents the i-th sampling point, θ i Indicates the angle corresponding to the i-th sampling point; N is the number of sampling points;

[0015] Step 2.4) Clustering of equidistant sampling points: For each set of equidistant sampling point positions and angles corresponding to the stop line mask obtained in step 2.3), first perform DBSCAN clustering on the angle set to obtain a clustered subset of sampling point angles with similar angles for the stop line mask. Then, the sampling points of each angle cluster subset are clustered based on the position, and several sampling point cluster subsets with similar positions under the angle subset are obtained. The angle and position of each sampling point in this subset are similar, representing a stop line instance, which is the set of equidistant sampling points corresponding to the instance;

[0016] Step 2.5) Stop line instantiation: Based on each sampling point subset obtained in step 2.4), calculate the instantiation description of each stop line, where the stop line attributes include length, direction, and center.

[0017] Preferably, the step 2.5) specifically comprises the following steps:

[0018] The location of each sampling point cluster subset is The corresponding angle is The average direction of the stop line is calculated according to formula (1); the projection length of the sampling point in this direction is calculated according to formula (2) as the length of the stop line; the position average of the sampling points corresponding to the maximum and minimum projection values ​​is calculated according to formula (3) and used as the center of the stop line; finally, the stop line instances whose length is less than the set threshold are deleted to obtain the stop line instance description in the single-frame bird's-eye view image. The specific calculation formula of the stop line parameters is as follows:

[0019]

[0020] L line =max 1≤i≤L (Pro(p i ,θ line ))-min 1≤i≤L (Pro(pi ,θ line )) (2)

[0021] C line =(p min_pro +p max_pro ) / twenty three) Where L represents the size of the subset; p min_pro and p max_pro They represent the original sampling points corresponding to the minimum and maximum projection values ​​respectively, and Pro(a,b) represents the projection value of position a on the unit vector b.

[0022] Preferably, the step 3) specifically includes:

[0023] Step 3.1) For the front-view image, perform point cloud projection based on the camera and lidar extrinsic parameter matrices to obtain the point cloud information corresponding to each instance mask in the front-view image obtained in step 2.2), and then obtain the 3D spatial representation of the instance; based on the current pose, convert the 3D spatial representation of each instance to the global coordinate system; compare with each object instance in the global instance map, calculate the point cloud geometric similarity and title feature cosine similarity, and weighted fusion according to the given weight; compare the instance fusion threshold, initialize or fuse the current instance object; object fusion involves updating the point cloud data and title features, and the update process is as follows: update the point cloud and corresponding bounding box by taking the point cloud union and re-denoising; update the object title feature by weighted fusion of features based on the number of observations; finally, construct a 3D global instance map of ground objects;

[0024] Step 3.2) For the bird's-eye view image, project the ground instance representation of the parking line obtained in step 2.5) onto the front view image based on the inverse perspective principle to obtain the pixel coordinates of the parking line in the front view image; calculate the current ground equation based on the laser point cloud data, and transform the ground equation into the camera coordinate system based on the extrinsic matrix between the camera and the lidar; transform the pixel coordinates of the parking line instance on the front view image obtained above into the normalized plane of the camera, and then project the coordinates on the normalized plane onto the ground equation to obtain the ground projection of the parking line instance, that is, the three-dimensional representation of the parking line in the current coordinate system, and transform it into the global coordinate system according to the current posture; calculate the angle difference Δθ between the parking line instance in the current frame and the parking line instance in the global instance map, and the projection overlap rate R o , vertical distance d from the center point c and the vertical distance d between the endpoints e When the angle difference and distance thresholds are less than the given thresholds and the overlap rate is greater than the given threshold, the parking line instance is fused and updated; otherwise, a new parking line instance is added to the global map.

[0025] Preferably, in step 3.2), the parking line instance fusion update method includes: updating the parking line instance angle based on the length weight, updating the parking line instance length based on the endpoint projection of the parking line instance, and updating the parking line center based on the parking line endpoints corresponding to the maximum and minimum projection values.

[0026] Preferably, the step 4) specifically includes:

[0027] Step 4.1) The parking space judgment criteria provided by the user are fed into the large language model, and parsed to obtain the target object category and judgment criteria required for parking space judgment;

[0028] In step 4.2), based on the object category, the global instance map containing both ground and aboveground objects obtained in step 3) is retrieved for a set of target instances of the target object category required in step 4.1). Based on the location of the target instance set in the map, the relationship information between each target instance and other related objects in its vicinity is obtained, forming a relationship description including position and direction. By combining these relationship descriptions with the parking space judgment criteria obtained in step 4.1), the large language model selects candidate parking space-related object combinations that meet the requirements.

[0029] In step 4.3), based on the candidate parking space-related object combination obtained in step 4.2), the center position and orientation of the object are first obtained based on the object's point cloud information. Then, according to the parking space judgment conditions obtained in step 4.1), the potential locations of parking spaces are analyzed. Then, based on the above potential locations, the length and width information of the parking spaces are treated as fixed values, and the center position and orientation information of the parking spaces that meet the conditions are calculated through geometric methods, finally obtaining the candidate parking spaces.

[0030] Preferably, in step 4.3), the candidate parking spaces obtained may overlap, and overlapping candidate parking spaces with similar directions are merged based on the size of the occupied areas; if the overlapping parking spaces are parallel parking spaces and inclined parking spaces, parallel parking spaces are given priority; finally, parking spaces with unreasonable occupation and those that are inconsistent with common sense are eliminated to obtain the final parking space detection result.

[0031] The present invention has the following beneficial effects:

[0032] This paper proposes an intelligent parking space detection method based on open set instance perception in an open environment, which improves the parking space recognition and detection capabilities in open environments. The method is based on a rich instance representation of the environment and uses a large language model to flexibly infer parking spaces. It has the flexibility to adapt to different parking space judgment criteria and is suitable for standard parking lots as well as non-standard parking environments where parking space lines are blurred, damaged, or non-existent. First, the system acquires environmental point clouds and forward-facing camera information, processes them to produce a bird's-eye view image, and fully perceives the ground and aboveground environment based on the forward-facing and bird's-eye view images. Open vocabulary detection and segmentation techniques are applied to the forward-facing and bird's-eye view images, enabling excellent open-set perception in open environments. Parking line masks are instantiated and post-processed to represent parking line instances as line segments, facilitating subsequent instance-based reasoning. Laser data is combined to establish a three-dimensional global open-set instance representation of the environment. Based on user-provided verbal descriptions, a large language model identifies parking space judgment objects and judgment conditions, enabling interactive parking space judgment based on verbal interaction. Object instances that meet the conditions are retrieved from the global map, and parking spaces are derived through geometric reasoning based on object relationships. Finally, the final parking space is selected based on parking space type priority and common sense knowledge about parking space existence, ensuring optimal and rational parking space detection. This method ultimately improves parking space recognition capabilities in open environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the parking space intelligent detection method based on open environment open set instance perception of the present invention;

[0034] Figure 2 A schematic diagram of a three-dimensional instance map construction method of the present invention;

[0035] Figure 3 A schematic diagram of the parking space case-based reasoning method of the present invention;

[0036] Figure 4 This is the parking space detection effect of the present invention in indoor standard parking scenarios and outdoor open parking scenarios. DETAILED DESCRIPTION

[0037] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0038] This implementation case uses a front-view camera and a 16-line laser radar to perceive the environment. The flow chart of the parking space intelligent detection method based on open environment open set instance perception is as follows: Figure 1 As shown in the figure, this method obtains multimodal environmental information by acquiring laser point clouds, front-view images, and bird's-eye view images; uses open-set environmental perception technology to identify parking space-related instances; combines laser data to establish a three-dimensional global instance; uses a large language model to identify parking space judgment conditions, and obtains the target parking space through geometric reasoning and common sense screening. The specific steps include the following:

[0039] 1) Obtain the environmental lidar point cloud and forward-looking camera information. For the forward-looking camera image, obtain the corresponding bird's-eye view image through inverse perspective mapping (IPM).

[0040] 2) Open vocabulary detection and segmentation technology is used on the front view image and the bird's-eye view image. Different prompt word technologies are used to identify instance masks related to vehicles, parking lines, etc., and the parking line masks are instantiated and post-processed to obtain a complete image instance description. The perception process can be referred to Figure 2 The first half of the process includes the following details:

[0041] 2.1) Instance Object Detection. The Grounding DINO model uses object category hints as input for ground object detection in front-view images, and predefined text description hints as input for ground object detection in bird's-eye view images. Ultimately, the object detection bounding boxes and confidence scores for the front-view and bird's-eye view images of the current frame are obtained.

[0042] 2.2) Instance Object Mask Segmentation: For the detection results of the front view image and bird's-eye view image obtained in step 2.1), the TAP (Tokenize Anything via Prompting) model is used to obtain the image mask, title, and other features corresponding to each detection result in the front view image and bird's-eye view image of the current frame.

[0043] 2.3) Post-processing of the stop line mask. Based on the instance object mask of the current frame obtained in step 2.2), the stop line mask is retrieved from the instance object list of each frame of the bird's-eye view according to the instance title cosine similarity, and the outer contour corresponding to the mask is obtained; each stop line mask outer contour is equidistantly sampled to obtain a set of equidistant sampling points. For each equidistant sampling point, the angle between the two adjacent sampling points is used as the angle value of the sampling point. The angle is limited to 0° to 180°, and the angle set corresponding to the equidistant sampling points is obtained.

[0044] 2.4) Clustering of equidistant sampling points. For each stop line mask obtained in step 2.3), the equidistant sampling point positions and angle sets corresponding to each stop line mask are first clustered using DBSCAN to obtain a subset of sampling point angles with similar angles for the stop line mask. Then, the sampling points of each angle cluster subset are clustered based on the position, and several sampling point cluster subsets with similar positions under the angle subset are obtained. The angle and position of each sampling point in this subset are similar, representing a stop line instance, which is the set of equidistant sampling points corresponding to the instance.

[0045] 2.5) Stop line instantiation. Based on each sampling point subset obtained in step 2.4), calculate the instantiation description of each stop line. The stop line attributes include length, direction, and center. The calculation formula is as follows. The specific steps are as follows: The position of each sampling point cluster subset is The corresponding angle is The average direction of the stop line is calculated using Formula 1. The projection length of the sampling point in this direction is calculated using Formula 2 as the length of the stop line. The average of the sampling points corresponding to the maximum and minimum projection values ​​is calculated using Formula 3 as the center of the stop line. Finally, the stop line instances with too small a length are deleted to obtain a description of the stop line instances in a single-frame bird's-eye view image. The specific calculation formula for the stop line parameters is as follows:

[0046]

[0047] L line =max 1≤i≤L (Pro(p i ,θ line ))-min 1≤i≤L (Pro(p i ,θ line )) (2)

[0048] C line =(p min_pro +p max_pro ) / twenty three)

[0049] where p min_pro and p max_pro They represent the original sampling points corresponding to the minimum and maximum projection values ​​respectively, and Pro(a,b) represents the projection value of position a on the unit vector b.

[0050] 3) Combine the open set instance perception results of the front view image and bird's-eye view image obtained in step 2) with the laser point cloud data to construct a 3D global instance map including the ground and above-ground objects. The map construction process can be referred to Figure 2 The second half includes the following detailed steps:

[0051] 3.1) For the front-view image, perform point cloud projection based on the camera and lidar extrinsic matrix to obtain the point cloud information corresponding to each instance mask in the front-view image obtained in step 2.2), thereby obtaining a 3D spatial representation of the instance. Based on the current pose, transform the 3D spatial representation of each instance to the global coordinate system. Compare the point cloud geometric similarity and title feature cosine similarity with each object instance in the global instance map, and perform a weighted fusion based on a given weight. Initialize or fuse the current instance object by comparing the instance fusion threshold. Object fusion involves updating the point cloud data and title features. The update process is as follows: update the point cloud and corresponding bounding box by taking the point cloud union and re-denoising. Update the object title features by weighted fusion based on the number of observations. Finally, construct a 3D global instance map of ground objects.

[0052] 3.2) For the bird's-eye view image, project the parking line ground instance representation obtained in step 2.5) onto the front view image through the inverse IPM transform to obtain the pixel coordinates of the parking line in the front view image; calculate the current ground equation based on the laser point cloud data, and transform the ground equation to the camera coordinate system based on the extrinsic parameter matrix between the camera and the lidar; transform the pixel coordinates of the parking line instance on the front view image obtained above to the camera normalized plane, and then project the coordinates on the normalized plane onto the ground equation to obtain the ground projection of the parking line instance, that is, the three-dimensional representation of the parking line in the current coordinate system, and transform it to the global coordinate system according to the current posture; calculate the angle difference Δθ between the parking line instance in the current frame and the parking line instance in the global instance map, and the projection overlap rate R o , vertical distance d from the center point c and the vertical distance d between the endpoints e When the angle difference and distance thresholds are less than a given threshold, and the overlap ratio is greater than a given threshold, a parking line instance fusion update is performed. Otherwise, a new parking line instance is added to the global map. The parking line update process is as follows: the parking line instance angle is updated based on the length weight, the parking line instance length is updated based on the projection of the parking line instance endpoints, and the parking line center is updated based on the parking line endpoints corresponding to the maximum and minimum projection values. Finally, a 3D global instance map of ground parking lines is constructed.

[0053] 4) Based on the parking space judgment conditions provided by the user, the large language model identifies the parking space judgment object type and parking space judgment criteria including distance and direction; and retrieves object instances that meet the conditions in the global instance map obtained in step 3), and uses geometric reasoning methods based on object relationships to obtain parking spaces; based on the priority of parking space types and common sense knowledge about parking space existence, the final parking space is selected. The parking space reasoning process can be referred to Figure 3 , the specific details include:

[0054] 4.1) The parking space judgment criteria provided by the user are fed into the large language model, and parsed to obtain the target object categories and judgment criteria required for parking space judgment. For example, if the user's judgment criteria is "a parking space may exist between two substantially parallel adjacent vehicles," the parsing yields "vehicle" as the detection object, and "two substantially parallel adjacent vehicles" as the judgment criteria. If the user's judgment criteria is "if there is a parked vehicle next to a wall, then other locations next to the wall can be used as parking spaces," the parsing yields "wall" and "vehicle" as the detection objects, and "there is a parked vehicle next to the wall" as the judgment criteria. If the user's judgment criteria is "two parking lines with a length of approximately 6 meters, a spacing of approximately 3 meters, and substantially parallel directions can form a parking space," the parsing yields "parking lines" as the detection object, and "a length of approximately 6 meters, a spacing of approximately 3 meters, and substantially parallel directions" as the judgment criteria.

[0055] 4.2) According to the object category, the target instance set of the target object category required in step 4.1) is retrieved from the global instance map containing ground and above-ground objects obtained in step 3), According to the location of the target instance set in the map, get each target instance Other related objects in its vicinity The relationship information of the two components is used to form a relationship description including position and direction. By combining these relationship descriptions with the parking space judgment conditions obtained in step 4.1), the large language model selects candidate parking space-related object combinations that meet the requirements.

[0056] 4.3) Based on the candidate parking space-related object combinations obtained in step 4.2), first determine the center position and orientation of the objects based on their point cloud information. Then, based on the parking space judgment criteria obtained in step 4.1), analyze the potential locations of parking spaces. For example, the potential parking space locations corresponding to the three cases mentioned in step 4.1 are "the line connecting two vehicles," "the straight line along the wall between vehicles," and "the line connecting the centers of the two parking lines." Based on these potential locations, treat the length and width of the parking spaces as fixed values ​​and use geometric methods to calculate the center position and orientation of the parking spaces that meet the criteria, ultimately obtaining candidate parking spaces.

[0057] 4.4) The candidate parking spaces obtained in step 4.3) above may overlap. Overlapping candidate parking spaces with similar orientations are merged based on their occupied area. If the overlapping spaces are parallel or angled, parallel spaces are prioritized. Finally, parking spaces with unreasonable occupancy or that are inconsistent with common sense, such as those located near a tree, are eliminated to obtain the final parking space detection results.

[0058] Figure 4Parking space inference performance is presented on the SUPS underground parking dataset and data collected from real vehicles in an open campus environment. In these two scenarios, numerous parking lines are obscured, blurred, or damaged, as well as in situations where parking lines are absent but many vehicles are still parked. Parking space detection results demonstrate that this method achieves robust parking space detection performance in scenarios with poor visibility, such as blurred or damaged parking lines, or even in scenarios where parking lines are absent. It also demonstrates good parking space inference performance in open scenes and non-standard parking environments.

[0059] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An intelligent parking space detection method based on open environment open set instance perception, characterized in that: include: Step 1) Obtain the environmental lidar point cloud and forward-looking camera information. For the forward-looking camera image, obtain the corresponding bird's-eye view image through inverse perspective mapping; Step 2) Open vocabulary detection and segmentation techniques are used on the front view image and the bird's-eye view image. Different prompt word techniques are used to identify instance masks related to parking spaces, including vehicles and stop lines. The stop line masks are then instantiated and post-processed to obtain a complete image instance description, which includes: Step 2.1) Instance Object Detection: Use the object category hint as the input to the Grounding DINO model for ground object detection in the front view image, and use the predefined text description hint as the input to the Grounding DINO model for ground object detection in the bird's-eye view image. Ultimately, the object detection boxes and confidence scores for the front view image and bird's-eye view image of the current frame are obtained. Step 2.2) Instance Object Mask Segmentation: For the detection results of the front view image and the bird's-eye view image obtained in step 2.1), use the TAP model to obtain the image mask and title features corresponding to each detection result in the front view image and the bird's-eye view image of the current frame; Step 2.3) Post-processing of the stop line mask: Based on the current frame instance object mask obtained in step 2.2), the stop line mask is retrieved from the mask's title features, and the outer contour corresponding to the mask is obtained; each stop line mask outer contour is equidistantly sampled to obtain a set of equidistant sampling point positions. For each equidistant sampling point, the angle between the two adjacent sampling points is used as the angle value of the sampling point. The angle is limited to 0° to 180°, and the angle set corresponding to the equidistant sampling points is obtained. Among them, p i represents the i-th sampling point, θ i Indicates the angle corresponding to the i-th sampling point; N is the number of sampling points; Step 2.4) Clustering of equidistant sampling points: For each set of equidistant sampling point positions and angles corresponding to the stop line mask obtained in step 2.3), first perform DBSCAN clustering on the angle set to obtain a clustered subset of sampling point angles with similar angles for the stop line mask. Then, the sampling points of each angle cluster subset are clustered based on the position, and several sampling point cluster subsets with similar positions under the angle subset are obtained. The angle and position of each sampling point in this subset are similar, representing a stop line instance, which is the set of equidistant sampling points corresponding to the instance; Step 2.5) Stop line instantiation: Based on each sampling point subset obtained in step 2.4), calculate the instantiation description of each stop line, where the stop line attributes include length, direction, and center; Step 3) combining the open set instance perception results of the front view image and the bird's-eye view image obtained in step 2) with the laser point cloud data to construct a three-dimensional global instance map including the ground and above-ground objects; In step 4, the large language model identifies the parking space judgment object type and parking space judgment criteria including distance and direction based on the parking space judgment conditions provided by the user. It then searches for object instances that meet the conditions in the global instance map obtained in step 3) and uses geometric reasoning based on object relationships to obtain parking spaces. The final parking space is selected based on the priority of parking space types and common sense knowledge about parking space existence.

2. The method for intelligent parking space detection based on open environment open set instance perception according to claim 1, characterized in that: The specific steps of step 2.5) include: The location of each sampling point cluster subset is The corresponding angle is The average direction of the stop line is calculated according to formula (1); the projection length of the sampling point in this direction is calculated according to formula (2) as the length of the stop line; the position average of the sampling points corresponding to the maximum and minimum projection values ​​is calculated according to formula (3) and used as the center of the stop line; finally, the stop line instances whose length is less than the set threshold are deleted to obtain the stop line instance description in the single-frame bird's-eye view image. The specific calculation formula of the stop line parameters is as follows: L line =max 1≤i≤L (Pro(p i ,i line ))-min 1≤i≤L (Pro(p i ,i line )) (2) C line =(p min_pro +p max_pro ) / 2 (3) Where L represents the size of the subset; p min_pro and p max_pro They represent the original sampling points corresponding to the minimum and maximum projection values ​​respectively, and Pro(a,b) represents the projection value of position a on the unit vector b.

3. The method for intelligent parking space detection based on open environment open set instance perception according to claim 1, characterized in that: The step 3) specifically includes: Step 3.1) For the front-view image, perform point cloud projection based on the camera and lidar extrinsic parameter matrices to obtain the point cloud information corresponding to each instance mask in the front-view image obtained in step 2.2), and then obtain the 3D spatial representation of the instance; based on the current pose, convert the 3D spatial representation of each instance to the global coordinate system; compare with each object instance in the global instance map, calculate the point cloud geometric similarity and title feature cosine similarity, and weighted fusion according to the given weight; compare the instance fusion threshold, initialize or fuse the current instance object; object fusion involves updating the point cloud data and title features, and the update process is as follows: update the point cloud and corresponding bounding box by taking the point cloud union and re-denoising; update the object title feature by weighted fusion of features based on the number of observations; finally, construct a 3D global instance map of ground objects; Step 3.2) For the bird's-eye view image, project the ground instance representation of the parking line obtained in step 2.5) onto the front view image based on the inverse perspective principle to obtain the pixel coordinates of the parking line in the front view image; calculate the current ground equation based on the laser point cloud data, and transform the ground equation into the camera coordinate system based on the extrinsic matrix between the camera and the lidar; transform the pixel coordinates of the parking line instance on the front view image obtained above into the normalized plane of the camera, and then project the coordinates on the normalized plane onto the ground equation to obtain the ground projection of the parking line instance, that is, the three-dimensional representation of the parking line in the current coordinate system, and transform it into the global coordinate system according to the current posture; calculate the angle difference Δθ between the parking line instance in the current frame and the parking line instance in the global instance map, and the projection overlap rate R o , vertical distance d from the center point c and the vertical distance d between the endpoints e When the angle difference and distance thresholds are less than the given thresholds and the overlap rate is greater than the given threshold, the parking line instance is fused and updated; otherwise, a new parking line instance is added to the global map.

4. The method for intelligent parking space detection based on open environment open set instance perception according to claim 3, characterized in that: In step 3.2), the parking line instance fusion update method includes: updating the parking line instance angle based on the length weight, updating the parking line instance length based on the endpoint projection of the parking line instance, and updating the parking line center based on the parking line endpoints corresponding to the maximum and minimum projection values.

5. The method for intelligent parking space detection based on open environment open set instance perception according to claim 1, characterized in that: The step 4) specifically includes: Step 4.1) The parking space judgment criteria provided by the user are fed into the large language model, and parsed to obtain the target object category and judgment criteria required for parking space judgment; In step 4.2), based on the object category, the global instance map containing both ground and aboveground objects obtained in step 3) is retrieved for a set of target instances of the target object category required in step 4.1). Based on the location of the target instance set in the map, the relationship information between each target instance and other related objects in its vicinity is obtained, forming a relationship description including position and direction. By combining these relationship descriptions with the parking space judgment criteria obtained in step 4.1), the large language model selects candidate parking space-related object combinations that meet the requirements. In step 4.3), based on the candidate parking space-related object combination obtained in step 4.2), the center position and orientation of the object are first obtained based on the object's point cloud information. Then, according to the parking space judgment conditions obtained in step 4.1), the potential locations of parking spaces are analyzed. Then, based on the above potential locations, the length and width information of the parking spaces are treated as fixed values, and the center position and orientation information of the parking spaces that meet the conditions are calculated through geometric methods, finally obtaining the candidate parking spaces.

6. The method for intelligent parking space detection based on open environment open set instance perception according to claim 5, characterized in that: In step 4.3), if there is overlap among the candidate parking spaces obtained, the overlapping candidate parking spaces with similar directions are merged according to the size of their occupied areas; If the overlapping parking spaces are parallel parking spaces and inclined parking spaces, the parallel parking spaces will be given priority; finally, the parking spaces that are unreasonably occupied and inconsistent with common sense are excluded to obtain the final parking space detection results.

Images