Image-based multi-level parking lot positioning method, device and car

Abstract

The present disclosure relates to the field of automobile positioning technology, and discloses a multi-layer parking lot positioning method and device based on images and an automobile. The method loads a map corresponding to the level of the multi-layer parking lot where the vehicle is currently located when the vehicle is in a parking lot positioning mode. The map is a map file established in advance by using images collected by the vehicle in the parking lot. Real-time images around the vehicle are obtained, and feature points of the real-time images are extracted. The feature points of the real-time images are matched with feature points in images saved in the map file, and an image with the most matching feature points with the real-time image is determined as a target image. The relative position relationship of the vehicle between the real-time image and the target image is calculated. Based on the relative position relationship and the pose of the vehicle in the target image, the positioning result of the vehicle in the parking lot at the corresponding time of the real-time image is determined. The present disclosure solves the problem of how to realize the positioning of the vehicle in the multi-layer parking lot at low cost.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle positioning technology, and in particular to an image-based multi-level parking lot positioning method, apparatus, and vehicle. Background Technology

[0002] In recent years, with rapid economic development and rising living standards, the number of cars on the road has increased significantly, leading to a surge in large, multi-story underground parking garages. However, these garages often suffer from complex road layouts and parking difficulties, severely impacting the parking experience and efficiency for drivers. Therefore, navigation systems within underground parking garages are receiving increasing attention.

[0003] However, current parking lot vehicle positioning solutions have many problems. For example, one existing technology uses QR codes installed on vehicles and cameras in the parking lot to identify these codes for continuous vehicle positioning. However, this technology requires modifications to both the vehicle and the parking lot, resulting in high costs. Another existing technology uses a positioning system based on a monocular camera and chassis information to provide vehicle positioning results on a two-dimensional parking lot plane. However, this technology cannot provide vertical positioning information, thus making cross-level positioning in multi-level parking lots impossible. Yet another existing technology provides a method for cross-level positioning in multi-level parking lots based on high-precision maps. This method loads high-precision maps of each level of the nearby parking lot by acquiring the vehicle's GPS location and then performs map matching to obtain the positioning result. However, in practical applications, the accuracy of GPS signals in parking lots is often low, frequently resulting in incorrect map loading. Furthermore, loading maps for each level simultaneously involves a large amount of data, leading to long loading times.

[0004] Therefore, it is evident that how to solve the problem of vehicle positioning in multi-story parking lots at low cost is a technical challenge currently encountered in automotive applications. Summary of the Invention

[0005] In view of the shortcomings of the prior art, this disclosure presents an image-based multi-level parking lot positioning method, apparatus and vehicle to solve the problem of how to achieve vehicle positioning in multi-level parking lots at low cost.

[0006] According to one aspect of the present disclosure, an image-based multi-level parking lot positioning method is provided, comprising: when a vehicle is in a parking lot positioning mode, loading a map corresponding to the level of the multi-level parking lot where the vehicle is currently located, wherein the map is a map file pre-built using images collected by the vehicle in the parking lot, wherein the map file includes images pre-collected by the vehicle in the parking lot, and the spatial coordinates of feature points in the images in the parking lot map coordinate system and the pose of the vehicle in the parking lot map coordinate system at the corresponding time of the image, wherein the parking lot where the vehicle is currently located includes a multi-level parking lot; acquiring real-time images around the vehicle and extracting feature points from the real-time images; matching the feature points of the real-time images with feature points in the images stored in the map file, and determining the image in the map file with the most matching feature points with the real-time images as the target image; calculating the relative positional relationship of the vehicle between the real-time images and the target image; and determining the positioning result of the vehicle in the parking lot at the corresponding time of the real-time images based on the relative positional relationship and the pose of the vehicle in the target image, wherein the positioning result includes the pose of the vehicle in the map coordinate system.

[0007] According to another aspect of the embodiments of this disclosure, an image-based multi-level parking lot positioning device is provided, comprising: a map loading module configured to load a map corresponding to the level of the multi-level parking lot where the vehicle is currently located when the vehicle is in parking lot positioning mode, the map being a map file pre-built using images collected by the vehicle in the parking lot, wherein the map file includes images pre-collected by the vehicle in the parking lot, and the spatial coordinates of feature points in the images in the parking lot map coordinate system and the vehicle's pose in the parking lot map coordinate system at the corresponding time of the image, the parking lot where the vehicle is currently located including multi-level parking lots; an image acquisition module configured to acquire real-time images around the vehicle and extract feature points from the real-time images; an image matching module configured to match feature points in the real-time images with feature points in images stored in the map file, and determine the image in the map file with the most matching feature points with the real-time images as the target image; a position calculation module configured to calculate the relative positional relationship between the real-time images and the target image; and a vehicle positioning module configured to determine the positioning result of the vehicle in the parking lot at the corresponding time of the real-time images based on the relative positional relationship and the vehicle's pose in the target image, the positioning result including the vehicle's pose in the map coordinate system.

[0008] According to another aspect of the present disclosure, an automobile is provided, comprising: one or more processors and a storage device for storing one or more programs, which, when executed by one or more processors, cause the automobile to perform the steps of the above-described method.

[0009] The beneficial effects of this disclosure are as follows: When the vehicle is in parking lot positioning mode, a map corresponding to the level of the multi-level parking lot where the vehicle is currently located is loaded. The map is a map file pre-built using images collected by the vehicle in the parking lot. The map file includes images pre-collected by the vehicle in the parking lot, the spatial coordinates of feature points in the images in the parking lot map coordinate system, and the vehicle's pose in the parking lot map coordinate system at the corresponding time. The parking lot where the vehicle is currently located includes multi-level parking lots. Real-time images around the vehicle are acquired, and feature points in the real-time images are extracted. The feature points in the real-time images are matched with the feature points in the images stored in the map file, and the image in the map file with the most matching feature points with the real-time images is determined as the target image. The relative positional relationship of the vehicle between the real-time images and the target image is calculated. Based on the relative positional relationship and the vehicle's pose in the target image, the positioning result of the vehicle in the parking lot at the corresponding time of the real-time image is determined. The positioning result includes the vehicle's pose in the map coordinate system. Thus, the vehicle can be positioned at any level in a multi-level parking lot using only the surrounding images, without the need to modify the parking lot, reducing the implementation cost of positioning. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0011] Figure 1 This is a flowchart of an image-based multi-level parking lot positioning method provided in an embodiment of this disclosure;

[0012] Figure 2 This is a schematic diagram of the vehicle positioning principle in mapping mode provided in this embodiment of the disclosure;

[0013] Figure 3 This is a schematic diagram of the structure of an image-based multi-level parking lot positioning device provided in an embodiment of this disclosure;

[0014] Figure 4 This is a schematic diagram of another image-based multi-level parking lot positioning device provided in an embodiment of this disclosure;

[0015] Figure 5 This is a schematic diagram of the structure of a computer system provided in an embodiment of this disclosure. Detailed Implementation

[0016] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the following embodiments and sub-samples in the embodiments can be combined with each other.

[0017] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. Therefore, the drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0018] Numerous details are explored in the following description to provide a more thorough explanation of embodiments of the present disclosure. However, it will be apparent to those skilled in the art that embodiments of the present disclosure may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present disclosure.

[0019] Unless otherwise stated, the term "multiple" means two or more.

[0020] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0021] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0022] Please see Figure 1 The diagram shows a flowchart of an image-based multi-level parking lot positioning method provided in an embodiment of this disclosure. Figure 1 The image-based multi-level parking lot localization method described in the text can be applied to vehicles and executed by the vehicles themselves. For example... Figure 1 As shown, the image-based multi-level parking lot localization method includes:

[0023] S101, when the vehicle is in parking lot positioning mode, load the map corresponding to the level of the multi-level parking lot where the vehicle is currently located. The map is a map file created in advance using images collected by the vehicle in the parking lot. The map file includes images collected in advance by the vehicle in the parking lot, the spatial coordinates of feature points in the images in the parking lot map coordinate system, and the vehicle's pose in the parking lot map coordinate system at the corresponding time of the image. The parking lot where the vehicle is currently located includes multi-level parking lots.

[0024] S102, acquire real-time images of the area around the vehicle and extract feature points from the real-time images;

[0025] S103, Match the feature points of the real-time image with the feature points of the image stored in the map file, and determine the image in the map file with the most matching feature points as the target image;

[0026] S104, calculate the relative positional relationship of vehicles between the real-time image and the target image;

[0027] S105, based on the relative positional relationship and the vehicle's pose in the target image, determine the vehicle's location in the parking lot at the corresponding moment in the real-time image. The location result includes the vehicle's pose in the map coordinate system.

[0028] The working principle of the above-mentioned image-based multi-level parking lot positioning method is as follows: When a vehicle enters the parking lot, the vehicle is switched to parking lot positioning mode. At this time, the user can input the current level of the vehicle on the vehicle's screen, thereby loading a map of the current parking lot level. Simultaneously, one or more cameras installed on the vehicle collect real-time images of the vehicle's surroundings and extract feature points from the real-time images. By matching the feature points in the real-time images with the pre-saved images in the map, the frame image with the most matching feature points is selected as the target image. Since the positions of the feature points are constant and the vehicle's pose in the target image is known, the vehicle's pose at the corresponding moment in the real-time image can be quickly calculated based on the relative positional relationship between the real-time image and the target image in the map, thus achieving vehicle positioning in the parking lot.

[0029] This embodiment loads a map corresponding to the level of the multi-level parking lot where the vehicle is currently located when the vehicle is in parking lot positioning mode. The map is a map file pre-built using images collected by the vehicle in the parking lot. The map file includes the pre-collected images of the vehicle in the parking lot, the spatial coordinates of feature points in the images in the parking lot map coordinate system, and the vehicle's pose in the parking lot map coordinate system at the corresponding time. The parking lot where the vehicle is currently located includes multi-level parking lots. Real-time images of the vehicle's surroundings are acquired, and feature points of the real-time images are extracted. The feature points of the real-time images are matched with the feature points in the images stored in the map file, and the image in the map file with the most matching feature points with the real-time images is determined as the target image. The relative positional relationship of the vehicle between the real-time images and the target image is calculated. Based on the relative positional relationship and the vehicle's pose in the target image, the positioning result of the vehicle in the parking lot at the corresponding time of the real-time image is determined. The positioning result includes the vehicle's pose in the map coordinate system. This allows the vehicle to be positioned at any level in a multi-level parking lot using only the surrounding images, without requiring modifications to the parking lot, thus reducing the implementation cost of vehicle positioning.

[0030] In one embodiment, loading a map corresponding to the level of the multi-story parking garage where the vehicle is currently located includes: obtaining the level of the current vehicle in the multi-story parking garage as input by the user, and loading a map corresponding to the level input by the user.

[0031] Specifically, when a vehicle first enters a parking lot, it needs to confirm its initial parking level. For example, when the vehicle switches to parking lot location mode, the vehicle's infotainment screen provides an input option to ask the user to confirm the current parking level of the vehicle. This allows the vehicle to automatically load the corresponding map based on the parking level entered by the user for vehicle location.

[0032] In one embodiment, loading the map corresponding to the current level of the multi-level parking lot where the vehicle is located includes: obtaining the vehicle's wheel speed and three-axis angular velocity after determining the current parking lot level; determining the vehicle's pitch angle and elevation change in the parking lot map coordinate system at the corresponding moment based on the vehicle's wheel speed and three-axis angular velocity; determining whether the vehicle is in a cross-level state based on the pitch angle and elevation change; if so, determining the parking lot level the vehicle is going to in conjunction with the current parking lot level, and loading the map of the parking lot level the vehicle will go to, wherein the cross-level state includes the vehicle entering the next parking level or entering the previous parking level.

[0033] The wheel speed of the vehicle can be either the rear wheel speed or the front wheel speed; this embodiment does not limit this. Furthermore, the triaxial angular velocity can be obtained by pre-installing an IMU (Inertial Measurement Unit) on the vehicle.

[0034] It's worth noting that the vehicle can use its three-axis angular velocity to calculate its trajectory. Based on the pitch angle and elevation changes calculated by the welding machine, it can determine whether the vehicle is within the parking lot's multi-level passageway. For example, when the vehicle begins to enter the passageway, the sign of the pitch angle can be used to determine whether the vehicle is heading to the upper or lower parking level. The vehicle then begins loading the map of the upper or lower parking level. Once the vehicle reaches the new level, it begins to locate itself using the loaded map.

[0035] It should be noted that in practical applications, the current level of a vehicle can also be determined by identifying feature points or specific markers in real-time images. However, due to the uncertainty of image recognition, the wheel speed and three-axis angular velocity of the vehicle are relatively more reliable and less prone to errors.

[0036] In one embodiment, calculating the relative positional relationship of the vehicle between the real-time image and the target image includes: calculating the vehicle pose change between the real-time image and the target image based on the feature points matched between the real-time image and the target image using epipolar constraint, PnP, or ICP algorithms.

[0037] Epipolar constraints refer to estimating camera motion based on multiple pairs of 2D pixels between two images, given the known 2D pixel coordinates.

[0038] Perspective-n-Point (PnP) is a method for solving the motion of 3D to 2D point pairs, aiming to determine the pose of the camera coordinate system relative to the world coordinate system. In this embodiment, the camera coordinate system can be the camera coordinate system used to acquire real-time images from the vehicle, while the world coordinate system can be the parking lot coordinate system. Specifically, based on the real-time images, the relative positional relationship between feature points and the vehicle can be determined. Then, by utilizing the change relationship between the vehicle coordinate system and the parking lot coordinate system, the pose of the vehicle in the parking lot coordinate system at the corresponding moment in the real-time image can be determined. Since the pose of the vehicle in the parking lot coordinate system in the target image is known, the relative positional relationship of the vehicle in the parking lot coordinate system between the real-time image and the target image can be calculated, i.e., the change in vehicle pose.

[0039] The ICP (Iterative Closest Point) algorithm is based on the EM (Expectation-Maximization) algorithm, using an alternating iterative method to optimize for the optimal value. ICP consists of two iterative optimization steps: optimizing point cloud matching and optimizing motion estimation. Point cloud matching involves placing points from two frames of point cloud data in the same coordinate system, finding the nearest point in the other frame, and defining this as a matching pair. Motion estimation involves constructing and solving least-squares equations based on the matching results from the two point cloud frames.

[0040] It is worth mentioning that the epipolar constraint, PnP, and ICP algorithms all utilize matching feature points to calculate motion estimates between two frames, i.e., changes in vehicle pose. Since these algorithms are standard in the field of visual motion, they will not be described in detail here.

[0041] This embodiment uses feature points matched between the real-time image and the target image. Based on epipolar constraints, PnP, and ICP algorithms, the relative positional relationship of the vehicle between the real-time image and the target image can be quickly calculated. Then, using the pose of the target image in the parking lot coordinate system (i.e., the map coordinate system), the two-dimensional pose of the vehicle in the parking lot coordinate system at the corresponding moment in the real-time image is calculated, thus completing the vehicle localization.

[0042] In one embodiment, a map file is pre-built using images collected by the vehicle in the parking lot, including: when the vehicle is in mapping mode, determining the parking lot level where the vehicle is currently located, and acquiring images of the vehicle's surroundings in real time; extracting feature points from the images, and matching the feature points in the current frame image with the feature points in the previous frame image to obtain matched feature points; determining the vehicle's pose and the spatial coordinates corresponding to the feature points based on the matched feature points; and saving the spatial coordinates corresponding to the feature points, the image where the feature points are located, and the vehicle's pose at the corresponding time as a map file of the parking lot levels traversed by the vehicle.

[0043] Specifically, when it is necessary to map a parking lot, the vehicle can be switched to mapping mode, and the user can input the current parking lot level of the vehicle on the vehicle's infotainment screen to complete the initialization of the vehicle's parking lot level.

[0044] Similar to the vehicle positioning mode in parking lots, the feature points extracted from real-time images and surrounding images can include FSAT corner points and BRIEF descriptors in each frame of the image.

[0045] FSAT corners are defined as follows: a pixel is likely a corner if it is located in a different region from a sufficient number of pixels in its neighborhood. In other words, certain attributes are distinct. Considering grayscale images, if the grayscale value of a pixel is greater or less than the grayscale values ​​of a sufficient number of pixels in its neighborhood, then that pixel is likely a corner.

[0046] The BRIEF descriptor, also known as BRIEF (Binary Robust Independent Elementary Features), describes detected feature points. It is a binary-coded descriptor that abandons the traditional method of describing feature points using region grayscale histograms, greatly accelerating the construction of feature descriptors and significantly reducing feature matching time. It is a very fast and promising algorithm.

[0047] Since BRIEF is merely a feature descriptor, the location of the feature points must be obtained beforehand. This can be achieved using algorithms such as FAST feature point detection, Harris corner detection, SIFT, and SURF. Then, the feature descriptor can be constructed using the feature point neighborhood BRIEF algorithm.

[0048] Specifically, matching feature points between different images (e.g., the previous frame in surrounding images and the current frame, or a real-time image and surrounding images) refers to accurately matching the same object in two images from different viewpoints. Although images exist in the form of grayscale matrices in computers, using only the grayscale values ​​of an image cannot accurately identify the same object in two images. This is because grayscale is affected by lighting, and the grayscale value of the same object changes as the image viewpoint changes. Therefore, it is necessary to find features that remain unchanged even when the camera moves and rotates (the viewpoint changes), and to use these invariant features to find the same object in images from different viewpoints.

[0049] This embodiment uses FSAT corner points and BRIEF descriptors as feature points of the image, which can preserve important features of the image, reduce the amount of image data computation, and achieve accurate matching of feature points between different images.

[0050] In one embodiment, feature points are extracted from the image, and feature points in the current frame image are matched with feature points in the previous frame image to obtain matched feature points. This includes: generating image pyramids for the current frame image and the previous frame image; extracting feature points from the image pyramids corresponding to the current frame image and the previous frame image layer by layer based on the gray-scale centroid method; and matching feature points in the current frame image and the previous frame image using the approximate nearest neighbor algorithm to obtain matched feature points.

[0051] Specifically, an image pyramid is an efficient yet conceptually simple structure for interpreting images at multiple resolutions. An image pyramid is a collection of images arranged in a pyramid shape with progressively decreasing resolution, all originating from the same original image.

[0052] Considering the FSAT corner points and BRIEF descriptors mentioned above, FAST corner points lack scale invariance. Therefore, a Gaussian pyramid is constructed, and FAST corner points are detected at each pyramid level. The gray-scale centroid method is used to calculate the orientation of the feature points. It is assumed that there is an offset between the gray level of the corner point and its centroid; this vector can represent a direction. The main idea of ​​this method is to first treat the neighborhood of the feature point as a patch, then calculate the centroid of this patch, and finally connect the centroid to the feature point, finding the angle between this line and the horizontal axis, which represents the orientation of the feature point.

[0053] In this embodiment, the approximate nearest neighbor algorithm is used to match feature points between two frames of images. The principle is to convert the two frames of images into feature vector spaces with FSAT corner points and BRIEF descriptors respectively, and then find the vector with the closest distance to the feature vector to match the feature points, thereby determining the matching feature points between the two frames of images.

[0054] Of course, in practical applications, other algorithms similar to the approximate nearest neighbor algorithm can also be used to match feature points. For example, the k nearest neighbor algorithm can also be used. This disclosure does not limit this.

[0055] In one embodiment, determining the vehicle pose and the spatial coordinates corresponding to the feature points based on the matched feature points includes: establishing a vehicle coordinate system; setting the vehicle coordinate system at the time corresponding to the first frame image at the start of mapping as the origin of the parking lot map coordinate system; calculating the vehicle pose change between the corresponding times of the two frames using epipolar constraint, PnP, or ICP algorithms based on the matched feature points in the two frames; and determining the vehicle pose relative to the parking lot map coordinate system at the time corresponding to the current frame image, and the spatial coordinates of the matched feature points relative to the parking lot map coordinate system, based on the vehicle pose at the time corresponding to the previous frame image and the vehicle pose change between the corresponding times of the two frames.

[0056] Please see Figure 2 This shows a schematic diagram of the vehicle's positioning principle in mapping mode, such as... Figure 2 As shown, a vehicle coordinate system X0Y can be established with the vehicle's longitudinal centerline as the X-axis, the rear wheel centerline as the Y-axis, and the rear wheel center as the origin. For example, Figure 2 The coordinates of feature point D in the vehicle coordinate system can be represented as (X, Y). Furthermore, in the vehicle mapping mode, Figure 2Image P1, acquired by the vehicle at time T1, can be a frame of surrounding image acquired by the vehicle, and image P2, acquired by the vehicle at time T2, can be another frame of surrounding image acquired by the vehicle.

[0057] Therefore, when vehicle mapping begins, the vehicle's coordinate system at the moment of the first frame of the acquired images surrounding the vehicle is used as the parking lot coordinate system. That is, the origin of the vehicle's coordinate system coincides with the origin of the parking lot map coordinate system, or it can also be called the map coordinate system. For example, Figure 2 In the image, the coordinates of feature point D in the surrounding image in the parking lot coordinate system can be represented as (X0, Y0).

[0058] Determining vehicle pose change based on feature points matched between two image frames refers to the relationship between the vehicle's motion position changes at different times corresponding to the two image frames. In other words, by utilizing the pose changes of the same feature points in different images, the motion changes of the camera position on the vehicle can be determined. Since the relative position of the camera and the vehicle is fixed, the pose change of the vehicle between corresponding times in two different image frames can be determined. Then, through the transformation relationship between the vehicle coordinate system and the parking lot coordinate system, the vehicle's pose in the parking lot coordinate system at the corresponding time in the current image frame and the spatial coordinates corresponding to the feature points in the current image frame can be obtained. Specifically, the vehicle's pose here includes the vehicle's position, for example, the vehicle's position in the parking lot coordinate system.

[0059] This embodiment extracts feature points from the image and uses feature point matching to calculate the spatial coordinates of the feature points in each frame of the image in the parking lot coordinate system, as well as the vehicle's pose. In other words, each frame of the image corresponds to a position marker of a vehicle in the parking lot, thereby saving the image, the spatial coordinates of the feature points in the image, and the vehicle's pose at the corresponding time in the image as a map file.

[0060] It is worth mentioning that the working principle of the vehicle in parking lot positioning mode and mapping mode is the same. Therefore, the principle of calculating the vehicle pose change between corresponding times of two frames using epipolar constraints, PnP or ICP algorithms in mapping mode can be referred to the description in parking lot positioning mode, and will not be repeated here.

[0061] In addition, the number of surrounding images acquired by the vehicle in mapping mode is very large, and the vehicle is not always in motion while in the parking lot. Therefore, if all the corresponding frames of surrounding images are saved as map files, the map files will occupy too much storage space or contain some duplicate images.

[0062] In one embodiment, after determining the vehicle pose and the spatial coordinates corresponding to the feature points based on the matched feature points, the method further includes: determining whether the current frame image meets one of the following conditions: the number of matched feature points between the current frame image and the previous frame image is less than or equal to a preset number threshold; the relative motion distance between the current frame image and the previous keyframe image is greater than a preset distance threshold; the time difference between the current frame image and the previous keyframe image is greater than a preset time threshold; if the current frame image meets at least one of the above conditions, then the current frame image is determined to be a keyframe image, wherein the previous frame image includes keyframe images, and the images stored in the map file include keyframe images.

[0063] Specifically, the aforementioned quantity threshold, distance threshold, and time threshold are all preset thresholds based on experience or experimental results. In practical applications, these thresholds can be fixed or adjusted according to different parking lots or vehicle data feedback. This disclosure does not impose any restrictions on this.

[0064] Specifically, multiple conditions are set here. An image can be used as a keyframe image for image processing if it meets any one of these conditions. Among these conditions, the previous frame image is the keyframe image; therefore, the first frame image can be defaulted to as the keyframe image at the start of map construction. This embodiment extracts feature points by selecting keyframe images from the acquired surrounding images. Therefore, it is best to save the images in the map file as keyframe images, thereby significantly reducing the number of images in the map file.

[0065] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0066] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0067] Figure 3 This is a schematic diagram of an image-based multi-level parking lot positioning device provided in an embodiment of this disclosure. Figure 3 As shown, the image-based multi-level parking lot positioning device includes:

[0068] The map loading module 301 is configured to load the map corresponding to the level of the multi-level parking lot where the vehicle is currently located when the vehicle is in parking lot positioning mode. The map is a map file created in advance using images collected by the vehicle in the parking lot. The map file includes images collected in advance by the vehicle in the parking lot, the spatial coordinates of feature points in the images in the parking lot map coordinate system, and the vehicle's pose in the parking lot map coordinate system at the corresponding time of the image. The parking lot where the vehicle is currently located includes multi-level parking lots.

[0069] The image acquisition module 302 is configured to acquire real-time images of the area around the vehicle and extract feature points from the real-time images;

[0070] The image matching module 303 is configured to match feature points of a real-time image with feature points of an image stored in a map file, and determine the image in the map file that has the most matching feature points with the real-time image as the target image.

[0071] The position calculation module 304 is configured to calculate the relative positional relationship of the vehicles between the real-time image and the target image;

[0072] The vehicle positioning module 305 is configured to determine the vehicle's positioning result in the parking lot at the corresponding moment in the real-time image based on the relative positional relationship and the vehicle's pose in the target image. The positioning result includes the vehicle's pose in the map coordinate system.

[0073] This embodiment loads a map corresponding to the level of the multi-level parking lot where the vehicle is currently located when the vehicle is in parking lot positioning mode. The map is a map file pre-built using images collected by the vehicle in the parking lot. The map file includes the pre-collected images of the vehicle in the parking lot, the spatial coordinates of feature points in the images in the parking lot map coordinate system, and the vehicle's pose in the parking lot map coordinate system at the corresponding time. The parking lot where the vehicle is currently located includes multi-level parking lots. Real-time images of the vehicle's surroundings are acquired, and feature points of the real-time images are extracted. The feature points of the real-time images are matched with the feature points in the images stored in the map file, and the image in the map file with the most matching feature points with the real-time images is determined as the target image. The relative positional relationship of the vehicle between the real-time images and the target image is calculated. Based on the relative positional relationship and the vehicle's pose in the target image, the positioning result of the vehicle in the parking lot at the corresponding time of the real-time image is determined. The positioning result includes the vehicle's pose in the map coordinate system. Thus, the vehicle can be positioned at any level in a multi-level parking lot using only the surrounding images, without the need to modify the parking lot, reducing the implementation cost of positioning.

[0074] In one embodiment, Figure 3 The map loading module 301 is used to obtain the level of the current vehicle in the multi-level parking lot as input by the user, and load the map corresponding to the level input by the user.

[0075] In one embodiment, Figure 3The map loading module 301 is used to obtain the vehicle's wheel speed and three-axis angular velocity when the current parking level of the vehicle is determined; based on the vehicle's wheel speed and three-axis angular velocity, determine the vehicle's pitch angle and elevation change in the parking map coordinate system at the corresponding moment in the real-time image; determine whether the vehicle is in a cross-level state based on the pitch angle and elevation change; if so, determine the parking level to which the vehicle will go in combination with the current parking level of the vehicle, and load the map of the parking level to which the vehicle will go. The cross-level state includes the vehicle entering the next parking level or entering the previous parking level.

[0076] In one embodiment, Figure 3 The position calculation module 304 is used to calculate the vehicle pose change between the real-time image and the target image based on the feature points matched between the real-time image and the target image, using epipolar constraints, PnP or ICP algorithms.

[0077] In one embodiment, see Figure 4 The image-based multi-level parking lot positioning device also includes:

[0078] The map building module 306 is configured to, when the vehicle is in mapping mode, determine the parking lot level where the vehicle is currently located and acquire images of the vehicle's surroundings in real time; extract feature points from the images and perform feature point matching between the feature points in the current frame and the feature points in the previous frame to obtain matched feature points; based on the matched feature points, determine the vehicle's pose and the spatial coordinates corresponding to the feature points; and save the spatial coordinates corresponding to the feature points, the image where the feature points are located, and the vehicle's pose at the corresponding time as a map file of the parking lot levels that the vehicle has passed through.

[0079] In one embodiment, see continue to see Figure 3 The image-based multi-level parking lot positioning device also includes:

[0080] The image filtering module 307 is configured to, after determining the vehicle's pose and the spatial coordinates corresponding to the feature points based on the matched feature points, determine whether the current frame image meets one of the following conditions: the number of matched feature points between the current frame image and the previous frame image is less than or equal to a preset threshold; the relative motion between the current frame image and the previous keyframe image is greater than a preset threshold; the time difference between the current frame image and the previous keyframe image is greater than a preset threshold; if the current frame image meets at least one of the above conditions, then the current frame image is determined to be a keyframe image, wherein the previous frame image includes keyframe images, and the images stored in the map file include keyframe images.

[0081] In one embodiment, the feature points include: FSAT corner points and / or BRIEF descriptors in each frame of the image.

[0082] In one embodiment, Figure 4 The map building module 306 is used to generate image pyramids of the current frame image and the previous frame image; based on the gray-scale centroid method, feature points of the image pyramids corresponding to the current frame image and the previous frame image are extracted layer by layer; the approximate nearest neighbor algorithm is used to match the feature points of the current frame image and the feature points of the previous frame image to obtain the matching feature points.

[0083] In one embodiment, Figure 4 The map building module 306 is used to establish the vehicle coordinate system; the vehicle coordinate system corresponding to the first frame image at the start of map building is set as the origin of the parking lot map coordinate system; based on the feature points matched between the two frames, the vehicle pose change between the corresponding times of the two frames is calculated using epipolar constraint, PnP or ICP algorithms; based on the vehicle pose at the corresponding time of the previous frame image and the vehicle pose change between the corresponding times of the two frames, the pose of the vehicle relative to the parking lot map coordinate system at the corresponding time of the current frame image, and the spatial coordinates of the matched feature points relative to the parking lot map coordinate system are determined.

[0084] Additionally, please see Figure 5 This illustration shows a schematic diagram of the structure of a computer system provided in an embodiment of the present disclosure. Figure 5 The computer system described above can be applied to automobiles as the vehicle's infotainment system to execute the image-based multi-level parking lot positioning method described above. For example... Figure 5 As shown, the computer system 500 includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 502 or programs loaded from storage portion 508 into Random Access Memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0085] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0086] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the various functions defined in the apparatus of this application.

[0087] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0088] In this embodiment, the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0089] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), graphics processing units (GPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0090] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated subsamples, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other subsamples, wholes, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, then the relevant parts can be referred to the description of the method section.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A multi-level parking lot positioning method based on images, characterized in that, include: When the vehicle is in parking lot positioning mode, load the map corresponding to the level of the multi-level parking lot where the vehicle is currently located. The map is a map file created in advance using images collected by the vehicle in the parking lot. The map file includes images collected in advance by the vehicle in the parking lot, the spatial coordinates of feature points in the images in the parking lot map coordinate system, and the vehicle's pose in the parking lot map coordinate system at the corresponding time of the image. The parking lot where the vehicle is currently located includes multi-level parking lots. Acquire real-time images of the area surrounding the vehicle and extract feature points from the real-time images; The feature points of the real-time image are matched with the feature points of the images stored in the map file, and the image in the map file with the most matching feature points with the real-time image is determined as the target image; Calculate the relative positional relationship of the vehicles between the real-time image and the target image; Based on the relative positional relationship and the vehicle's pose in the target image, the positioning result of the vehicle in the parking lot at the corresponding time in the real-time image is determined, and the positioning result includes the vehicle's pose in the map coordinate system. This includes creating map files in advance using images collected from vehicles in parking lots, including: When the vehicle is in mapping mode, determine the current parking lot level of the vehicle and acquire images of the vehicle's surroundings in real time. Extract feature points from the image, and perform feature point matching between the feature points in the current frame image and the feature points in the previous frame image to obtain the matched feature points; Based on the matched feature points, the vehicle pose and the spatial coordinates corresponding to the feature points are determined. Save the spatial coordinates corresponding to the feature point, the image where the feature point is located, and the vehicle pose at the corresponding time in the image as a map file of the parking lot level that the vehicle passes through. After determining the vehicle's pose and the corresponding spatial coordinates based on the matched feature points, the method further includes: Determine if the current frame image meets one of the following conditions: The number of feature points matched between the current frame image and the previous frame image is less than or equal to a preset threshold. The relative motion between the current frame image and the previous keyframe image is greater than a preset threshold; The time difference between the current frame image and the previous keyframe image is greater than a preset threshold. If the current frame image satisfies at least one of the above conditions, then the current frame image is determined to be a keyframe image, wherein the previous frame image includes the keyframe image, and the image stored in the map file includes the keyframe image.

2. The image-based multi-level parking lot positioning method according to claim 1, characterized in that, The process of loading the map corresponding to the current level of the multi-story parking lot where the vehicle is located includes: obtaining the level of the current vehicle in the multi-story parking lot input by the user, and loading the map corresponding to the level input by the user. Alternatively, given the current parking level of the vehicle, obtain the vehicle's wheel speed and three-axle angular velocity; Based on the vehicle's wheel speed and three-axis angular velocity, determine the vehicle's pitch angle and elevation change in the parking lot map coordinate system at the corresponding moment in the real-time image; Based on the pitch angle and elevation changes, it is determined whether the vehicle is in a cross-level state. If so, the parking level to which the vehicle is going is determined by combining the current parking level of the vehicle, and the map of the parking level to which the vehicle will go is loaded. The cross-level state includes the vehicle entering the next parking level or entering the previous parking level.

3. The image-based multi-level parking lot positioning method according to claim 1, characterized in that, The calculation of the relative positional relationship of the vehicles between the real-time image and the target image includes: Based on the feature points matched between the real-time image and the target image, the vehicle pose change between the real-time image and the target image is calculated using epipolar constraint, PnP, or ICP algorithms.

4. The image-based multi-level parking lot positioning method according to claim 1, characterized in that, The feature points include: FSAT corner points and / or BRIEF descriptors in each frame of the image.

5. The image-based multi-level parking lot positioning method according to claim 4, characterized in that, The step of extracting feature points from the image and matching the feature points in the current frame with those in the previous frame to obtain matched feature points includes: Generate an image pyramid of the current frame image and the previous frame image; Based on the gray-scale centroid method, feature points of the image pyramid corresponding to the current frame image and the previous frame image are extracted layer by layer. The approximate nearest neighbor algorithm is used to match the feature points of the current frame image with the feature points of the previous frame image to obtain the matching feature points.

6. The image-based multi-level parking lot positioning method according to claim 5, characterized in that, The determination of the vehicle's pose and the spatial coordinates corresponding to the feature points based on the matched feature points includes: Establish the vehicle coordinate system; Set the vehicle coordinate system at the moment corresponding to the first frame of the image at the start of mapping as the origin of the parking lot map coordinate system; Based on the feature points matched between two frames of images, the vehicle pose change between corresponding times of the two frames of images is calculated using epipolar constraint, PnP or ICP algorithms. Based on the vehicle pose at the corresponding moment of the previous frame in the two frames and the change in vehicle pose between the corresponding moments of the two frames, the pose of the vehicle relative to the parking lot map coordinate system at the corresponding moment of the current frame is determined, as well as the spatial coordinates of the matched feature points relative to the parking lot map coordinate system.

7. A multi-level parking lot positioning device based on images, characterized in that, include: The map loading module is configured to load a map corresponding to the level of the multi-level parking lot where the vehicle is currently located when the vehicle is in parking lot positioning mode. The map is a map file pre-built using images collected by the vehicle in the parking lot. The map file includes images pre-collected by the vehicle in the parking lot, the spatial coordinates of feature points in the images in the parking lot map coordinate system, and the vehicle's pose in the parking lot map coordinate system at the corresponding time of the image. The parking lot where the vehicle is currently located includes multi-level parking lots. The image acquisition module is configured to acquire real-time images of the area around the vehicle and extract feature points from the real-time images; The image matching module is configured to match feature points of the real-time image with feature points of the images stored in the map file, and determine the image in the map file that has the most matching feature points with the real-time image as the target image; The position calculation module is configured to calculate the relative positional relationship of the vehicle between the real-time image and the target image; The vehicle positioning module is configured to determine the positioning result of the vehicle in the parking lot at the corresponding time of the real-time image based on the relative positional relationship and the vehicle's pose in the target image, wherein the positioning result includes the vehicle's pose in the map coordinate system. This includes creating map files in advance using images collected from vehicles in parking lots, including: When the vehicle is in mapping mode, determine the current parking lot level of the vehicle and acquire images of the vehicle's surroundings in real time. Extract feature points from the image, and perform feature point matching between the feature points in the current frame image and the feature points in the previous frame image to obtain the matched feature points; Based on the matched feature points, the vehicle pose and the spatial coordinates corresponding to the feature points are determined. Save the spatial coordinates corresponding to the feature point, the image where the feature point is located, and the vehicle pose at the corresponding time in the image as a map file of the parking lot level that the vehicle passes through. After determining the vehicle's pose and the corresponding spatial coordinates based on the matched feature points, the method further includes: Determine if the current frame image meets one of the following conditions: The number of feature points matched between the current frame image and the previous frame image is less than or equal to a preset threshold. The relative motion between the current frame image and the previous keyframe image is greater than a preset threshold; The time difference between the current frame image and the previous keyframe image is greater than a preset threshold. If the current frame image satisfies at least one of the above conditions, then the current frame image is determined to be a keyframe image, wherein the previous frame image includes the keyframe image, and the image stored in the map file includes the keyframe image.

8. A car, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by the one or more processors, cause the vehicle to perform the method as described in any one of claims 1 to 6.

Images