Parking space information determination method and device, electronic equipment and storage medium

Abstract

The invention provides a parking space information determination method, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes of automatic driving, automatic parking and the like. According to the specific implementation scheme, obstacle information and parking space information in the environment where a vehicle is located are determined according to an image collected by a collection device of the vehicle; according to the obstacle information and the parking space information, the parking space state is determined, and the parking space state indicates whether an obstacle exists in the area where the parking space is located; and in response to an obstacle existing in the parking space area, calling a large language model to generate parking space available information according to the obstacle information of the parking space area. The invention further provides a parking space information determination device, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models and other technical fields, and can be applied to scenarios such as autonomous driving and automatic parking. More specifically, the present disclosure provides a method, device, electronic device, storage medium and computer program product for determining parking space information. Background Art

[0002] In the automatic parking system, the determination of whether a parking space is available (parking determination) is a crucial component. Its role is to identify suitable parking locations in the parking lot and help the automatic parking system complete the parking operation accurately and safely. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device, storage medium and computer program product for determining parking space information.

[0004] According to a first aspect, a method for determining parking space information is provided, the method comprising: determining obstacle information and parking space information in an environment where a vehicle is located based on an image captured by a capturing device of the vehicle; determining a parking space status based on the obstacle information and the parking space information, wherein the parking space status indicates whether there is an obstacle in the area where the parking space is located; and in response to the presence of an obstacle in the area where the parking space is located, calling a large language model to generate parking space availability information based on the obstacle information in the parking space area.

[0005] According to a second aspect, a parking space information determination device is provided, the device comprising: an information determination module, for determining obstacle information and parking space information in the environment where the vehicle is located based on an image collected by a collection device of the vehicle; a parking space status determination module, for determining the parking space status based on the obstacle information and the parking space information, wherein the parking space status indicates whether there is an obstacle in the area where the parking space is located; and a parking space availability determination module, for generating parking space availability information by calling a large language model based on the obstacle information of the parking space area in response to the presence of an obstacle in the area where the parking space is located.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an exemplary system architecture to which a method and apparatus for determining parking space information can be applied according to an embodiment of the present disclosure;

[0012] Figure 2 is a flow chart of a method for determining parking space information according to an embodiment of the present disclosure;

[0013] Figure 3 is a system diagram of a method for determining parking space information according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram of a method for visually correcting a fisheye image according to an embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram of a method for performing depth estimation on a planar projection image according to an embodiment of the present disclosure;

[0016] Figure 6 is a schematic diagram of a method for determining parking space information according to an embodiment of the present disclosure;

[0017] Figure 7 is a block diagram of a parking space information determination device according to an embodiment of the present disclosure;

[0018] Figure 8 The present invention is a block diagram of an electronic device for a method for determining parking space information according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0020] Automatic parking systems usually include three main steps: parking space detection, parking space availability determination, and parking position estimation. Among them, parking space availability determination is a difficult point, because parking spaces in parking lots are often occupied by various debris or other obstacles, making it difficult for traditional parking space availability determination methods to cope with complex parking environments.

[0021] The traditional method of determining the availability of parking spaces includes obtaining parking space location information using a parking space detection model, obtaining obstacle spatial location and size information using an obstacle detection model, and completing the parking space availability determination through rule determination. Rule determination can include finding the parking space location using the parking space detection results obtained by the parking space detection model, and determining whether there are obstacles in the parking space location in combination with the 3D detection frame or occupancy results obtained by the obstacle detection. If there are obstacles, the parking space is generally determined to be unavailable.

[0022] Common parking lot debris includes shopping carts, trash cans, fallen leaves, snow, and occasional gravel. Some debris (such as shopping carts and trash cans) may occupy part or all of the parking space, affecting the availability of the parking space; while other debris (such as fallen leaves and light snow) have little impact on the parking process, or can even be ignored, despite their presence in the parking space. However, traditional parking space availability determination methods have difficulty distinguishing these different types of debris, and often make misjudgments, resulting in inaccurate parking space determination.

[0023] The above parking space availability determination method belongs to the traditional mode of combining a small amount of training data with a small vehicle-side model, which cannot take advantage of the massive data of autonomous driving scenarios. At the same time, in terms of data, the existing manual labeling method is inefficient, expensive, and has a long cycle, making it difficult to produce large amounts of data.

[0024] In addition, there is a high probability of encountering long-tail situations in parking scenarios, especially in outdoor parking lots, such as cans, plastic bags, stones, abandoned furniture, etc. The traditional solution cannot effectively identify long-tail obstacles, so it is impossible to use effective logical rules for subsequent judgment.

[0025] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0026] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0027] Figure 1 is a schematic diagram of an exemplary system architecture to which a method and apparatus for determining parking space information can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0028] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptops, etc.

[0030] The parking space information determination method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the parking space information determination device provided in the embodiment of the present disclosure can generally be set in the server 105. The parking space information determination method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the parking space information determination device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0031] Figure 2 is a flowchart of a method for determining parking space information according to an embodiment of the present disclosure.

[0032] like Figure 2 As shown, the parking space information determination method 200 includes operations S210 to S240.

[0033] In operation S210, obstacle information and parking space information in the environment where the vehicle is located are determined based on the image collected by the collection device of the vehicle.

[0034] The vehicle's data collection device can be a fisheye camera installed on the vehicle body. Fisheye cameras can be installed on the front, back, left, and right sides of the vehicle to obtain fisheye images of the front, back, left, and right sides of the vehicle body. By processing the collected fisheye images, obstacle information and parking space information around the vehicle body can be obtained.

[0035] For example, the fisheye image can be converted into a plane image, and key point detection can be performed in the plane image to obtain information such as the location, shape, and size of the obstacle. The plane image can also be reconstructed in three dimensions to obtain point cloud data, and then the location, shape, and size of the obstacle can be obtained. In addition, target detection and segmentation can be performed on the fisheye image or the plane image to obtain the detection frame and category of each obstacle in the image. For example, the categories can include shopping carts, trash cans, fallen leaves, snow, etc.

[0036] For example, the fisheye image can be converted into a bird's-eye view, and key point detection can be performed in the bird's-eye view to obtain the location information of the parking space, such as the coordinates of the four corner points of the parking space.

[0037] In operation S220, the parking space status is determined according to the obstacle information and the parking space information, wherein the parking space status indicates whether there is an obstacle in the area where the parking space is located.

[0038] After obtaining information such as the position, type and size of obstacles in the vehicle's environment and the position information of parking spaces, it is possible to determine whether there are obstacles in the area where each parking space is located based on the positional relationship between the obstacles and the parking spaces.

[0039] For example, the position information of the obstacle may include the three-dimensional spatial position information and the detection frame, and the position information of the parking space may include the three-dimensional spatial position information and the detection frame. The obstacle detection frame and the parking space detection frame may be superimposed on the image, and the image may be a fisheye image, or a plane image obtained by processing the fisheye image. In the image after superimposing the parking space detection frame and the obstacle detection frame, it may be determined whether each parking space detection frame overlaps with the obstacle detection frame. If there is an overlap, it may be determined that there is an obstacle in the area where the parking space is located. If there is no overlap, it may be determined that there is no obstacle in the area where the parking space is located.

[0040] When it is determined that there are no obstacles in the area where the parking space is located, it can be determined that the parking space is available and the automatic parking system can continue to perform the parking operation.

[0041] In operation S230 , in response to the presence of an obstacle in the parking space area, a large language model is called to generate parking space availability information according to obstacle information in the parking space area.

[0042] The parking space availability information may include the availability determination result of the parking space (or the parking availability determination result). When it is determined that there are obstacles in the parking space area, the powerful reasoning ability of the large model can be used to further determine whether the obstacles in the parking space area will constitute a substantial obstacle to vehicle parking based on the parking space information and obstacle information, thereby making an accurate parking space availability determination result.

[0043] For example, prompt information can be generated according to the prompt word template (Prompt) based on the parking space information and obstacle information. The prompt word template can also include vehicle information (such as chassis height), content requirements (such as requiring suggestions or reasoning processes, etc.) and format requirements, as well as examples, etc. The prompt information is input into the large language model, and the large language model can generate a judgment result on whether the parking space is available according to the requirements.

[0044] For example, the large model can determine the texture, shape, size and other characteristics of the obstacle based on its category and size information, and judge whether the obstacle will hinder the parking of the vehicle. For large obstacles such as trash cans, the large model will identify them as objects that need to be avoided. For fallen leaves or snow, the large model can determine that they will not pose a substantial obstacle to vehicle parking, thereby making an accurate judgment on the availability of parking spaces.

[0045] In the embodiment of the present disclosure, when it is determined that there are obstacles in the area where the parking space is located based on the obstacle information and the parking space information, a large language model is called to generate parking space availability information based on the obstacle information and the parking space information, which can improve the accuracy of the parking space availability information of the automatic parking system in complex environments.

[0046] The disclosed embodiment calls upon a large cloud model to determine parking spaces. The combination of the vehicle and the cloud maintains good real-time performance and computing efficiency, can adapt to a variety of parking scenarios, and can be applied on a large scale. It not only improves the intelligence level of the automatic parking system, but also provides solid technical support for the development of future smart parking lots.

[0047] According to an embodiment of the present disclosure, the category and size of the obstacle are determined; the physical information of the vehicle is determined; prompt information is obtained based on the category and size of the obstacle and the physical information of the vehicle; and a large language model is called to generate parking space availability information based on the prompt information.

[0048] The embodiments of the present disclosure may also input the physical information of the vehicle into the large language model, and the large language model may combine the obstacle information, the parking space information and the physical information of the vehicle to comprehensively determine whether the parking space is available. For example, the physical information of the vehicle may be the chassis height of the vehicle, and based on the chassis height and the height of the obstacle, it may be determined whether the vehicle can pass over the obstacle.

[0049] According to an embodiment of the present disclosure, whether an obstacle needs to be avoided is determined based on at least one of the type and size of the obstacle; in response to the obstacle not needing to be avoided, parking space availability information is generated based on the relationship between the chassis height and the maximum height of the obstacle's size.

[0050] For example, if the obstacle is a fallen leaf, snow, etc., it can generally be determined that the obstacle does not need to be avoided. However, if the fallen leaf or snow is large in size and occupies a large space in the parking space, it can be determined that the obstacle needs to be avoided.

[0051] If the obstacle does not need to be avoided, it can be determined whether the vehicle chassis height is greater than the maximum height of the obstacle. If so, it can be determined that the vehicle can pass over the obstacle and a parking space is available. Otherwise, it is determined that the vehicle cannot pass over the obstacle and a parking space is unavailable. For example, the large model can determine that the vehicle can pass over thin snow but cannot pass over rocks and other scenarios.

[0052] The embodiments of the present disclosure comprehensively judge whether a parking space is available by combining obstacle information, parking space information and vehicle physical information, thereby improving the accuracy of the parking space availability judgment.

[0053] Figure 3 It is a system diagram of a method for determining parking space information according to an embodiment of the present disclosure.

[0054] like Figure 3 As shown, this embodiment includes an obstacle information determination module 310, a parking space information determination module 320, a rule determination module 330 and a large model determination module 340. The obstacle information determination module 310 may include a visual correction submodule 311, a 3D structure estimation submodule 312 and an open set detection and segmentation submodule 313.

[0055] The following describes each submodule in the obstacle information determination module 310 .

[0056] The visual correction submodule 311 is used to convert the fisheye image into a coordinate system to obtain a plane projection image. The fisheye image is collected by a fisheye camera installed on the vehicle body. The fisheye camera has a certain pitch angle and is generally taken from a top-down perspective relative to the vehicle. Therefore, the fisheye image is an image of a top-down perspective relative to the vehicle coordinate system. For a more intuitive view, the fisheye image can be converted into a plane projection image based on the main vehicle coordinate system as a head-up perspective.

[0057] The 3D structure estimation submodule 312 is used to perform depth estimation on the plane projection image to obtain the depth information of each pixel in the plane projection image; determine the point cloud data of the plane projection image based on the depth information of each pixel in the plane projection image; and determine the position information of obstacles in the plane projection image based on the point cloud data.

[0058] After obtaining the plane projection image, the depth estimation model can be used to perform depth estimation on the plane projection image to obtain the depth information of the plane projection image. The three-dimensional spatial position information of each pixel point in the plane projection image can be determined based on the depth information. Point cloud data can be generated based on the three-dimensional spatial position information, and the three-dimensional position information of each obstacle can be determined in the point cloud data.

[0059] The open set detection and segmentation submodule 313 is used to detect and segment the image to obtain the detection frame and category of the obstacle, where the category includes multiple preset categories; and for obstacles in the image that do not belong to the preset category, determine the guidance information based on the detection frame of the obstacle, and call the multimodal large model based on the guidance information to obtain the category of the obstacle.

[0060] The open set detection and segmentation submodule 313 may include a target detection module and a SAM (Segment Anything Model) large model. The target detection model may be trained using training data in a closed set, and the categories of the training data in the closed set belong to multiple preset categories, such as pedestrians, vehicles, cones, trash cans, etc. commonly seen in parking scenes. By using the target detection model to detect and segment the image, the detection frame and category of obstacles belonging to the preset multiple categories can be detected from the image.

[0061] In parking scenarios, there is a high probability that obstacles will have a long tail. The long tail of obstacles refers to obstacles that appear infrequently but will continue to appear over time. For example, pedestrians, vehicles, cones, etc. are obstacles that appear frequently in parking scenarios, while shopping carts, animals, etc. are obstacles that appear infrequently and are long-tail obstacles. In outdoor parking scenarios, there is an even higher probability that obstacles will have a long tail, such as cans, plastic bags, stones, discarded furniture, and other long-tail obstacles.

[0062] In order to handle these unknown or new categories that did not appear in the training stage of the target detection model, the embodiments of the present disclosure use the SAM large model to detect and segment the image to obtain the detection box and category information of obstacles of new categories other than the preset categories.

[0063] The SAM model is a multimodal model that can include a visual processing model and a natural language processing model. It can process the input image based on the guidance information to determine the location and category of obstacles in the image. For example, the guidance information can be a detection box of an unknown category of obstacles in the image. The guidance information can also include text, such as a description text such as "This is an image of a parking lot".

[0064] The images processed by the above-mentioned target detection model and the SAM large model can be fisheye images, or plan views, top views, etc. after the fisheye images are processed.

[0065] The open set detection and segmentation submodule of the embodiment of the present disclosure detects obstacles belonging to preset categories in the image by using the target detection model, determines guidance information for obstacles that do not belong to the preset category based on the detection frame of the obstacle, and calls the SAM large model based on the guidance information to determine the category of the obstacle. It can effectively handle the long-tail phenomenon of obstacles, accurately identify the obstacle category, and improve the parking success rate.

[0066] Thus, the obstacle information determination module 310 can obtain the three-dimensional spatial position information, detection frame and category information of the obstacle.

[0067] The parking space information determination module 320 is used to convert the fisheye images in multiple directions into the top views in multiple directions; splice the top views in multiple directions to obtain a panoramic top view; and determine the position information of at least one parking space from the panoramic top view.

[0068] For example, the image collected by the vehicle may include fisheye images of the front, back, left, and right sides of the vehicle body. Each fisheye image can be converted into a top view through inverse perspective transformation. The four top views can be spliced ​​to obtain a 360-degree panoramic top view. Key point detection is performed in the panoramic top view to obtain a detection frame of at least one parking space. Key points may include, for example, the four corner points of the parking space. According to the conversion relationship between the top view coordinate system and the world coordinate system, the position information of the parking space in the top view can be converted into the position information in space. The above conversion relationship can be obtained by calibration of the spatial coordinates of known points (such as ground points). In this way, the parking space detection frame and the three-dimensional spatial position can be obtained.

[0069] After obtaining the obstacle information determined by the obstacle information determination module 310 and the parking space information determined by the parking space information determination module 320, the obstacle information and the parking space information can be input into the rule determination module 330. The rule determination module 330 determines whether there is an obstacle in the parking space based on the positional relationship between the obstacle and the parking space. If there is no obstacle in the parking space, it can be directly determined that the parking space is available and parking can continue.

[0070] When the rule determination module 330 determines that there is an obstacle in the parking space, in order to avoid misjudgment, the obstacle information and the parking space information can be provided to the large model determination module 340, and the large model determination module 340 further determines whether the obstacle in the parking space constitutes a substantial obstacle to the parking of the vehicle, thereby making accurate parking space availability information 350, which can include a parking availability determination result of the parking space.

[0071] The embodiments of the present disclosure combine a multimodal large model based on open set detection and a large language model to determine whether a parking space is available. This can effectively solve the problem of inaccurate parking space availability determination caused by long-tail obstacles, complex parking scenarios, etc., and improve the parking success rate.

[0072] Figure 4 is a schematic diagram of a method for visually correcting a fisheye image according to an embodiment of the present disclosure.

[0073] According to an embodiment of the present disclosure, the fisheye image is mapped onto a cylindrical surface in a virtual camera coordinate system to obtain a cylindrical projection image, wherein the virtual camera coordinate system is parallel to the vehicle coordinate system; and the cylindrical projection image is projected onto a plane perpendicular to the ground to obtain a plane projection image.

[0074] like Figure 4 As shown, the fisheye image 410 is an image captured by the vehicle body camera, which is a top-down perspective relative to the main vehicle coordinate system and has strong distortion. In order to adapt to the subsequent depth estimation process, the fisheye image 410 can be converted into a plan view based on the main vehicle coordinate system as a head-up perspective.

[0075] For example, the fisheye image can be first mapped onto a cylindrical surface 421 to obtain a cylindrical projection image. Since the cylindrical surface 421 has an opening angle of 180 degrees, mapping the fisheye image 410 onto the cylindrical surface 421 can reduce image loss. The cylindrical surface 421 can be presented by a constructed virtual camera, and the viewing angle of the virtual camera can be parallel to the driving direction of the vehicle, that is, the coordinate system of the virtual camera is parallel to the coordinate system of the host vehicle. This mapping method helps to provide a distortion-free wide-angle view in the horizontal direction.

[0076] Next, in order to convert the cylindrical projection image into a plane image based on the main vehicle coordinate system as a head-up view, the image pixels on the cylindrical surface 421 can be mapped to a plane perpendicular to the ground to obtain a plane projection image 431, thereby simulating the visual effect of looking forward horizontally from the perspective of the vehicle. Through such processing, the image with strong distortion originally captured by the fisheye lens is converted into a more intuitive, easy to understand and analyze head-up view, providing the vehicle with comprehensive, distortion-free visual information of the surrounding environment.

[0077] Figure 4 The cylindrical surface 422 in the fisheye camera coordinate system is also shown. The fisheye camera coordinate system is a top-down perspective relative to the vehicle coordinate system. The image blind area is smaller in the top-down perspective, but the imaging effect is not good. For example, the plane view projected from the cylindrical surface 422 is a plane projection image 432. The ground objects (such as the trees marked by the red circle) in the plane projection image 432 are tilted, and the horizon (such as the red line) is a curve.

[0078] The plane projection image 431 has a better imaging effect: objects on the ground (such as the trees marked by the red circle) conform to the vertical downward visual effect, and the horizon (such as the red line) appears as a straight line in the picture.

[0079] The embodiment of the present disclosure maps the fisheye image to a cylindrical surface in a virtual camera coordinate system parallel to the main vehicle coordinate system to obtain a cylindrical projection image, and then maps the cylindrical projection image to a plane perpendicular to the ground to obtain a plane projection image. This can obtain a more intuitive head-up view and present more comprehensive and distortion-free visual information.

[0080] Figure 5 is a schematic diagram of a method for performing depth estimation on a planar projection image according to an embodiment of the present disclosure.

[0081] like Figure 5 As shown, image 510 is a plane projection image, and depth estimation is performed on plane projection image 510 to obtain depth map 520. The depth information of each pixel in depth map 520 may be a relative depth. After each pixel in depth map 520 is depth-converted by depth conversion module 530, the absolute depth of each pixel is obtained. According to the absolute depth information, depth map 520 may be converted into point cloud image 540 representing the position in three-dimensional space.

[0082] The depth of the plane projection image 510 can be estimated using a pre-trained depth estimation model to obtain a depth map 520. The relative depth information of each pixel in the depth map 520 represents the relative distance of the object, not the actual distance from the shooting source (fisheye camera). For example, the relative depth of the depth map 520 increases along the direction indicated by the arrow, which can indicate that the object represented by the pixels above the depth map 520 is relatively farther from the shooting source than the object represented by the pixels below.

[0083] In order to obtain the three-dimensional position information of the obstacle in the plane projection image 510, it is necessary to convert the relative depth in the depth image 520 into an absolute depth. The absolute depth is, for example, the vertical distance between the pixel point and the shooting source.

[0084] According to an embodiment of the present disclosure, a depth estimation is performed on a plane projection image to obtain a relative depth of each pixel in the plane projection image; a plurality of ground pixel points in the plane projection image are determined; the absolute depth of the ground pixel points is calculated based on the distance of an acquisition device from the ground; a conversion coefficient between the relative depth and the absolute depth is calibrated based on the relative depth and absolute depth of the plurality of ground pixel points; and the absolute depth of each pixel in the plane projection image is determined based on the calibrated conversion coefficient.

[0085] The depth conversion module 530 shows the geometric relationship of the camera model. The camera is, for example, a fisheye camera mounted on the vehicle body. In this model, the relative depth between the camera and the object can be described by a cylindrical surface. For example, the vertical distance between a point G' on the cylindrical imaging plane mapped to by the camera and the camera is the relative depth of the point. In other words, the absolute depth of each pixel point in the depth map 520 is the vertical distance from each pixel point on the cylindrical imaging plane to the camera in the geometric relationship of the camera model.

[0086] In order to convert the relative depth value into a more practical absolute depth value, the embodiment of the present disclosure calibrates the conversion coefficient between the relative depth and the absolute depth based on the prior information of the ground points combined with the geometric relationship of the camera model, and converts the relative depth of each pixel point into the absolute depth according to the calibrated conversion coefficient.

[0087] For example, the installation position of the camera on the vehicle body is known in advance, so the distance H of the camera from the ground is determined, that is, the height H of the ground point from the camera can be used as the prior information of the ground point. In addition, the angle between the line connecting the camera to the ground point and the vertical line from the camera to the ground can also be determined according to the position and angle of the camera installation. Therefore, in the geometric relationship of the camera model, the height H of the ground point G from the camera, and the angle between the line connecting the camera to the ground point G and the vertical line from the camera to the ground are determined, thereby determining the vertical distance from the ground point G to the camera, that is, the absolute depth of the ground point G.

[0088] The ground point detection algorithm can detect multiple ground points, and the absolute depth information of each ground point can be determined based on the distance between the camera and the ground and the geometric relationship of the camera model. The conversion coefficient between relative depth and absolute depth can be calibrated based on the absolute depth information of multiple ground points.

[0089] The conversion relationship between relative depth and absolute depth is as follows:

[0090] (1)

[0091] Where d represents the absolute depth, Indicates relative depth, s and t are conversion coefficients.

[0092] Using the relative depth and absolute depth of multiple ground points, multiple sets of equations for solving (s, t) can be constructed, and the optimal (s, t) that meets the constraints can be calculated using the least squares method. All pixel points (including ground points and non-ground points) are transformed using formula (1), thereby achieving the mapping of the entire scene from relative depth to absolute depth.

[0093] The embodiments of the present disclosure calibrate the conversion coefficient between relative depth and absolute depth based on prior information of ground points combined with the geometric relationship of the camera model, and convert the relative depth of each pixel point into absolute depth according to the calibrated conversion coefficient, thereby achieving accurate mapping from relative depth to actual three-dimensional spatial position and improving the accuracy of determining the three-dimensional position information of obstacles.

[0094] Figure 6 It is a schematic diagram of a method for determining parking space information according to an embodiment of the present disclosure.

[0095] like Figure 6 As shown, the fisheye image 601 may be a fisheye image taken by four fisheye cameras at the front, back, left, and right sides of the vehicle body. For the fisheye image, operations S611 to S612 are performed to obtain the three-dimensional position information of the obstacle.

[0096] In operation S611, the fisheye image is cylindrically projected to obtain a cylindrical projection image, and then the cylindrical projection image is planarly projected to obtain a planar projection image.

[0097] For example, performing cylindrical projection on the fisheye image may include projecting the fisheye image onto a cylindrical surface in a virtual camera coordinate system parallel to the vehicle coordinate system. Performing planar projection on the cylindrical projection image may include mapping the cylindrical projection image onto a plane perpendicular to the ground.

[0098] In operation S612, the planar projection image is three-dimensionally reconstructed to obtain point cloud data.

[0099] For example, three-dimensional reconstruction of a planar projection image may include performing depth estimation on the planar projection image, obtaining relative depth information of each pixel in the planar projection image, converting the relative depth into absolute depth, and obtaining point cloud data of each pixel based on the absolute depth.

[0100] By performing operations S621 and S622 on the fisheye image, a three-dimensional detection frame and category of the obstacle can be obtained.

[0101] In operation S621, the vehicle pixels are removed from the fisheye image to obtain a scene image containing only environment pixels.

[0102] In operation S622, open set detection and segmentation are performed on the scene graph to obtain detection frames and categories of obstacles in the environment.

[0103] For example, open set detection and segmentation of a scene graph may include using a target detection model to detect the position and category of a target object (obstacle) in the scene graph, and obtaining a detection frame and category of the obstacle. The categories that the target detection model can detect are multiple categories preset during the training process. For obstacles that have not appeared during the training process, the detection frame of the obstacle can be used as guidance information to call a multimodal large model for detection and segmentation to obtain the category of the obstacle.

[0104] For the fisheye image, operations S631 to S632 are performed to obtain a parking space detection frame.

[0105] In operation S631, the fisheye image is converted into a top view.

[0106] For example, each fisheye image may be subjected to an inverse perspective transformation to obtain a corresponding bird's-eye view, and the bird's-eye views in four directions may be spliced ​​to obtain a panoramic bird's-eye view.

[0107] In operation S632, a parking space detection is performed in the panoramic bird's-eye view to obtain a parking space frame.

[0108] For example, the four corner points of the parking space may be detected in the panoramic bird's-eye view to obtain a parking space frame.

[0109] According to an embodiment of the present disclosure, the location information of the parking space and the location information of the obstacle are superimposed on the environmental data to obtain a scene graph; and the state of the parking space is determined according to the positional relationship between the parking space and the obstacle in the scene graph.

[0110] In operation S640, the detection frame and category of the obstacle obtained in the open set detection and segmentation step and the parking space frame obtained in the parking space detection step are superimposed to obtain a superimposed image.

[0111] The superimposed image can intuitively show the positional relationship between the parking space and the obstacle, so as to determine whether there is an obstacle in the parking space.

[0112] In operation S650, rule determination is performed according to the superposition result of the obstacle and the parking space and the three-dimensional position information of the obstacle in the three-dimensional reconstruction result.

[0113] Based on the superposition results, it is possible to determine whether there are obstacles in the parking space area, the position of the obstacle in the parking space, etc., and based on the three-dimensional position information of the obstacle, it is possible to determine the size of the space occupied by the obstacle in the parking space.

[0114] Whether parking can continue can be determined according to certain rules based on whether there are obstacles in the parking space, the location of the obstacles in the parking space, and the size of the space occupied by the obstacles. If yes, parking can continue, otherwise the large model reasoning step of operation S660 is executed.

[0115] For example, if there is an obstacle in the parking space, the obstacle is located in the middle of the parking space, or the proportion of the space occupied by the obstacle in the parking space is greater than a threshold (for example, 10%), the parking space can be determined to be unavailable. If there is no obstacle in the parking space, or the obstacle is located at the edge of the parking space and the proportion of the space occupied by the obstacle in the parking space is less than a threshold, the parking space can be determined to be available.

[0116] If the parking space is determined to be available based on the rules, the parking can continue. If the parking space is determined to be unavailable based on the rules, operation S660 is performed for the large model reasoning step.

[0117] In operation S660 , available information of the parking space is output based on the obstacle information and the parking space information using the large language model.

[0118] For example, prompt information can be generated based on obstacle information, parking space information, and vehicle physical information, and the prompt information can be input into the large language model, which generates parking space availability information. The parking space availability information can include continue parking, honk reminder, no parking, switch to the next parking space, etc.

[0119] Table 1 below is an example of a prompt word template for a large language model.

[0120] Table 1

[0121] You need to assist the vehicle to complete the automatic parking task. The following is the input information: Vehicle: Vehicle chassis height 0.2 meters Object information in the parking space: Object 1: Category {water pipe}, maximum height {0.05 meters}, size {length 2.1 meters, width 0.5 meters} Object 2: Category {plant}, maximum height {0.1 meters}, size {length 0.5 meters, width 0.5 meters}. Please judge whether the parking space is suitable for parking based on the above information, and select an appropriate operation suggestion from the following options: Honk to remind: This item indicates that parking is allowed, but there are pedestrians walking nearby, and you need to pay attention to the movement of pedestrians; No parking, switch to the next parking space: This item indicates that there are immovable and unpassable obstacles in the parking space, and parking is not allowed; Continue parking: This item indicates that parking is allowed in the parking space; Driver intervention, {Reason for intervention}: The model cannot handle the situation, for example, the following situations occur: 1) The information provided is not sufficient to make a safe judgment; 2) There are movable obstacles that the driver needs to move away; 3) Other unknown situations, please explain the detailed reasons. The 3D information of the object given here may not be accurate enough. Please make a judgment based on the category information. For example, fallen leaves can generally be pressed over (even if the 3D information shows that they cannot be pressed over).

[0122] Table 2 below is an example of a large language model’s response.

[0123] Table 2

[0124] According to the information provided: the chassis height of the vehicle is 0.2 meters. There are two objects in the parking space: Object 1: The category is a water pipe, the maximum height is 0.05 meters, and the dimensions are 2.1 meters long and 0.5 meters wide. Object 2: The category is a plant, the maximum height is 0.1 meters, and the dimensions are 0.5 meters long and 0.5 meters wide. In order to determine whether parking is possible, we need to consider the relationship between the height of the ego vehicle chassis and the maximum height of the objects. The height of the ego vehicle chassis is 0.2 meters, while the maximum height of the water pipe is 0.05 meters and the maximum height of the plant is 0.1 meters. This means that when all objects are considered, the height of the ego vehicle chassis is greater than the height of any single object plus the potential gap between them (considering the actual size of the objects and the height of the ego vehicle, in fact, the height of the ego vehicle chassis is enough to pass through these objects without touching them). Therefore, based on the above analysis, parking is possible. So, the correct answer is: continue parking.

[0125] The embodiment of the present disclosure uses a large language model to determine whether a parking space is available. It can not only give a judgment result on whether a parking space is available, but also provide information such as operation suggestions and reasons for unavailability, thereby improving the intelligence level of the automatic parking system.

[0126] According to an embodiment of the present disclosure, the present disclosure also provides a parking space information determination device.

[0127] Figure 7 is a block diagram of a parking space information determination device according to an embodiment of the present disclosure.

[0128] like Figure 7 As shown, the parking space information determination device 700 includes an information determination module 710 , a parking space status determination module 720 and a parking space availability determination module 730 .

[0129] The information determination module 710 is used to determine obstacle information and parking space information in the environment where the vehicle is located based on the images collected by the collection device of the vehicle.

[0130] The parking space status determination module 720 is used to determine the parking space status according to the obstacle information and the parking space information, wherein the parking space status indicates whether there is an obstacle in the area where the parking space is located.

[0131] The parking space availability determination module 730 is used to generate parking space availability information by calling a large language model according to obstacle information of the parking space area in response to the presence of obstacles in the parking space area.

[0132] The parking space availability determination module 730 includes an obstacle information determination unit, a prompt information determination unit and an availability information generation unit.

[0133] The obstacle information determination unit is used to determine the type and size of the obstacle.

[0134] The physical information determination unit is used to determine physical information of the vehicle.

[0135] The prompt information determination unit is used to obtain prompt information according to the type and size of the obstacle and the physical information of the vehicle.

[0136] The available information generating unit is used to call the large language model to generate parking space available information based on the prompt information.

[0137] The physical information includes chassis height. The available information generation unit includes an avoidance determination subunit and an available information generation subunit.

[0138] The avoidance determination subunit is used to determine whether an obstacle needs to be avoided according to at least one of the type and size of the obstacle.

[0139] The available information generating subunit is used for generating parking space available information according to the relationship between the chassis height and the maximum height of the obstacle size in response to the obstacle not needing to be avoided.

[0140] According to an embodiment of the present disclosure, the image includes a fisheye image. The information determination module 710 includes a conversion unit, a depth estimation unit, a point cloud determination unit, and an obstacle position information determination unit.

[0141] The conversion unit is used to convert the coordinate system of the fisheye image to obtain a plane projection image.

[0142] The depth estimation unit is used to perform depth estimation on the plane projection image to obtain the depth information of each pixel in the plane projection image.

[0143] The point cloud determination unit is used to determine the point cloud data of the plane projection image according to the depth information of each pixel point in the plane projection image.

[0144] The obstacle position information determination unit is used to determine the position information of the obstacle in the plane projection image according to the point cloud data.

[0145] The conversion unit includes a cylindrical projection unit and a planar projection unit.

[0146] The cylindrical projection unit is used to map the fisheye image onto a cylindrical surface in a virtual camera coordinate system to obtain a cylindrical projection image, wherein the virtual camera coordinate system is parallel to the vehicle coordinate system.

[0147] The plane projection unit is used to project the cylindrical projection image onto a plane perpendicular to the ground to obtain a plane projection image.

[0148] The depth estimation unit includes a relative depth determination subunit, a ground pixel point determination subunit, a first absolute depth determination subunit, a calibration subunit and a second absolute depth determination subunit.

[0149] The relative depth determination subunit is used to perform depth estimation on the plane projection image to obtain the relative depth of each pixel in the plane projection image.

[0150] The ground pixel point determination subunit is used to determine a plurality of ground pixel points in the plane projection image.

[0151] The first absolute depth determination subunit is used to calculate the absolute depth of ground pixel points according to the distance between the acquisition device and the ground.

[0152] The calibration subunit is used to calibrate the conversion coefficient between the relative depth and the absolute depth according to the relative depth and the absolute depth of a plurality of ground pixels.

[0153] The second absolute depth determination subunit is used to determine the absolute depth of each pixel point in the plane projection image according to the calibrated conversion coefficient.

[0154] The point cloud determination unit is used to determine the point cloud data of the plane projection image according to the absolute depth of each pixel point in the plane projection image.

[0155] The information determination module 710 includes an obstacle detection unit and a category determination unit.

[0156] The obstacle detection unit is used to detect and segment the image to obtain the detection frame and category of the obstacle, wherein the category includes multiple preset categories.

[0157] The category determination unit is used to determine guidance information according to the detection frame of the obstacle for obstacles in the image that do not belong to the preset category, and to call the multimodal large model according to the guidance information to obtain the category of the obstacle.

[0158] The information determination module 710 includes a top view determination unit, a splicing unit, and a parking space detection unit.

[0159] The top view determining unit is used to convert the fisheye images of multiple orientations into top views of multiple orientations.

[0160] The stitching unit is used to stitch the top views from multiple directions to obtain a panoramic top view.

[0161] The parking space detection unit is used to determine the position information of at least one parking space from the panoramic bird's-eye view.

[0162] The parking space status determination module 720 includes an environment data determination unit, a scene graph determination unit and a parking space status determination unit.

[0163] The environment data determination unit is used to remove the vehicle data from the image to obtain the environment data of the environment where the vehicle is located.

[0164] The scene graph determination unit is used to superimpose the location information of the parking space and the location information of the obstacle into the environmental data to obtain a scene graph.

[0165] The parking space state determination unit is used to determine the parking space state according to the positional relationship between the parking space and obstacles in the scene graph.

[0166] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0167] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0168] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0169] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0170] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the parking space information determination method. For example, in some embodiments, the parking space information determination method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the parking space information determination method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the parking space information determination method in any other appropriate manner (e.g., by means of firmware).

[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0173] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0175] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0176] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0177] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0178] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining parking space information, comprising: Determining obstacle information and parking space information in the environment where the vehicle is located based on the image collected by the collection device of the vehicle; Determine a parking space status according to the obstacle information and the parking space information, wherein the parking space status indicates whether there is an obstacle in the area where the parking space is located; as well as In response to the presence of an obstacle in the area where the parking space is located, a large language model is called to generate parking space availability information according to obstacle information in the parking space area.

2. The method according to claim 1, wherein: The generating parking space availability information by calling a large language model according to the obstacle information of the parking space area includes: determining the type and size of the obstacle; determining physical information of the vehicle; Obtaining prompt information according to the type and size of the obstacle and the physical information of the vehicle; and The large language model is called to generate parking space availability information based on the prompt information.

3. The method according to claim 2, wherein: The physical information includes chassis height; and the calling of the large language model to generate parking space availability information based on the prompt information includes: Determining whether the obstacle needs to be avoided according to at least one of the type and size of the obstacle; In response to the obstacle not needing to be avoided, parking space availability information is generated according to a relationship between the chassis height and a maximum height of the obstacle sizes.

4. The method according to claim 1, wherein: The image includes a fisheye image; and the determining of obstacle information and parking space information in the environment where the vehicle is located based on the image collected by the collection device of the vehicle includes: Performing coordinate system conversion on the fisheye image to obtain a plane projection image; Performing depth estimation on the planar projection image to obtain depth information of each pixel in the planar projection image; Determining point cloud data of the plane projection image according to the depth information of each pixel point in the plane projection image; and The position information of the obstacle in the planar projection image is determined according to the point cloud data.

5. The method according to claim 4, wherein: The coordinate system conversion of the fisheye image to obtain a plane projection image comprises: Mapping the fisheye image onto a cylindrical surface in a virtual camera coordinate system to obtain a cylindrical projection image, wherein the virtual camera coordinate system is parallel to the vehicle coordinate system; and The cylindrical projection image is projected onto a plane perpendicular to the ground to obtain the plane projection image.

6. The method according to claim 4, wherein: The performing depth estimation on the planar projection image to obtain depth information of each pixel point in the planar projection image comprises: Performing depth estimation on the planar projection image to obtain a relative depth of each pixel in the planar projection image; Determine a plurality of ground pixel points in the plane projection image; Calculating the absolute depth of the ground pixel point according to the distance between the acquisition device and the ground; Calibrate a conversion coefficient between the relative depth and the absolute depth according to the relative depths and the absolute depths of the plurality of ground pixel points; and The absolute depth of each pixel in the plane projection image is determined according to the calibrated conversion coefficient.

7. The method according to claim 6, wherein: Determining the point cloud data of the plane projection image according to the depth information of each pixel point in the plane projection image comprises: The point cloud data of the plane projection image is determined according to the absolute depth of each pixel point in the plane projection image.

8. The method according to claim 1, wherein: Determining obstacle information and parking space information in the environment where the vehicle is located based on the image collected by the collection device of the vehicle includes: Detecting and segmenting the image to obtain a detection frame and a category of the obstacle, wherein the category includes a plurality of preset categories; and For obstacles in the image that do not belong to the preset category, guidance information is determined according to the detection frame of the obstacle, and the multimodal large model is called according to the guidance information to obtain the category of the obstacle.

9. The method according to claim 1, wherein: The image includes fisheye images in multiple directions; the determining of obstacle information and parking space information in the environment where the vehicle is located based on the image collected by the collection device of the vehicle includes: Convert the fisheye images in multiple directions into top views in multiple directions; splicing the top views from the multiple directions to obtain a panoramic top view; and The position information of at least one parking space is determined from the panoramic bird's-eye view.

10. The method according to claim 1, wherein: The determining the parking space status according to the obstacle information and the parking space information includes: Removing vehicle data from the image to obtain environmental data of the environment in which the vehicle is located; superimposing the location information of the parking space and the location information of the obstacle onto the environmental data to obtain a scene graph; and The parking space state is determined according to the positional relationship between the parking space and the obstacle in the scene graph.

11. A parking space information determination device, comprising: An information determination module, used to determine obstacle information and parking space information in the environment where the vehicle is located based on the image collected by the vehicle's collection device; A parking space status determination module, used to determine the parking space status according to the obstacle information and the parking space information, wherein the parking space status indicates whether there is an obstacle in the area where the parking space is located; as well as The parking space availability determination module is used to generate parking space availability information by calling a large language model according to obstacle information of the parking space area in response to the presence of obstacles in the parking space area.

12. The device according to claim 11, wherein The parking space availability determination module includes: An obstacle information determination unit, used to determine the type and size of the obstacle; a physical information determination unit, configured to determine physical information of the vehicle; a prompt information determining unit, configured to obtain prompt information according to the type and size of the obstacle and the physical information of the vehicle; and The available information generating unit is used to call the large language model to generate parking space available information based on the prompt information.

13. The device according to claim 12, wherein: The physical information includes chassis height; the available information generating unit includes: an avoidance determination subunit, configured to determine whether the obstacle needs to be avoided according to at least one of the type and size of the obstacle; The available information generating subunit is used for generating parking space available information according to the relationship between the chassis height and the maximum height of the obstacle size in response to the obstacle not needing to be avoided.

14. The device according to claim 11, wherein: The image includes a fisheye image; the information determination module includes: A conversion unit, used for performing coordinate system conversion on the fisheye image to obtain a plane projection image; A depth estimation unit, used to perform depth estimation on the plane projection image to obtain depth information of each pixel in the plane projection image; a point cloud determination unit, configured to determine point cloud data of the plane projection image according to depth information of each pixel point in the plane projection image; and The obstacle position information determining unit is used to determine the position information of the obstacle in the planar projection image according to the point cloud data.

15. The device according to claim 14, wherein: The conversion unit comprises: a cylindrical projection unit, configured to map the fisheye image onto a cylindrical surface in a virtual camera coordinate system to obtain a cylindrical projection image, wherein the virtual camera coordinate system is parallel to the vehicle coordinate system; and The plane projection unit is used to project the cylindrical projection image onto a plane perpendicular to the ground to obtain the plane projection image.

16. The device according to claim 14, wherein: The depth estimation unit comprises: A relative depth determination subunit, used to perform depth estimation on the plane projection image to obtain a relative depth of each pixel point in the plane projection image; A ground pixel point determination subunit, used to determine a plurality of ground pixel points in the plane projection image; A first absolute depth determination subunit, configured to calculate the absolute depth of the ground pixel point according to the distance between the acquisition device and the ground; a calibration subunit, configured to calibrate a conversion coefficient between the relative depth and the absolute depth according to the relative depths and the absolute depths of the plurality of ground pixel points; and The second absolute depth determination subunit is used to determine the absolute depth of each pixel in the plane projection image according to the calibrated conversion coefficient.

17. The device according to claim 16, wherein: The point cloud determination unit is used to determine the point cloud data of the plane projection image according to the absolute depth of each pixel point in the plane projection image.

18. The device according to claim 11, wherein The information determination module comprises: an obstacle detection unit, configured to detect and segment the image to obtain a detection frame and a category of the obstacle, wherein the category includes a plurality of preset categories; and The category determination unit is used to determine guidance information according to the detection frame of the obstacle in the image that does not belong to the preset category, and to call the multimodal large model according to the guidance information to obtain the category of the obstacle.

19. The device according to claim 11, wherein: The image includes fisheye images in multiple orientations; The information determination module comprises: A top view determining unit, used for converting the fisheye images of the multiple orientations into top views of the multiple orientations; a splicing unit, configured to splice the top views from the multiple directions to obtain a panoramic top view; and The parking space detection unit is used to determine the position information of at least one parking space from the panoramic bird's-eye view.

20. The device according to claim 11, wherein The parking space status determination module includes: An environmental data determination unit, used to remove the vehicle data from the image to obtain environmental data of the environment in which the vehicle is located; a scene graph determining unit, configured to superimpose the location information of the parking space and the location information of the obstacle onto the environmental data to obtain a scene graph; and A parking space state determination unit is used to determine the parking space state according to the positional relationship between the parking space and the obstacle in the scene graph.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images