Automatic parking mapping and positioning system and method

Abstract

The present invention discloses a system and method for automatic parking mapping and positioning, comprising an image information receiving module for receiving continuous image information of a vehicle's surrounding environment captured by at least two monocular cameras, a vehicle information assembly interface for receiving driving data information from a vehicle, a processing unit for generating feature points with depth information in each frame of the continuous image information of the vehicle's surrounding environment, and building a three-dimensional point cloud map after matching the feature points with the vehicle's driving data information, and a storage module for storing the three-dimensional point cloud map, wherein when the vehicle is repositioned, the processing unit can receive real-time image information of the vehicle's surrounding environment captured by the at least two monocular cameras in real time according to the image information receiving module, and use the image information to compare the stored three-dimensional point cloud map to determine the positioning posture of the at least two monocular cameras.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for automatic parking mapping and positioning. Background Art

[0002] With the development of in-vehicle artificial intelligence, the demand for automatic parking functions of autonomous vehicles has increased, especially the automatic parking technology in specific parking lots. The most important thing for automatic parking is to locate and build a map of the parking lot. Outdoor areas are usually located by the global satellite positioning system (GPS), but the relative accuracy is poor. As for indoor parking lots, GPS information cannot be located due to the obstruction of buildings. Therefore, the simultaneous localization and mapping technology is used to realize vehicle positioning and build a parking lot map in a visual recognition manner. The general practice is that the autonomous driving vehicle uses a single monocular camera to create a point cloud map that matches the real environment through computer vision on the path of the vehicle in the parking lot. When performing the subsequent automatic parking function, the vehicle's own position in the map can be determined by establishing the map. However, such a map often produces poor recognition results for the actual positioning posture of the vehicle, especially for parking lots without specific structures or low scene variability. Summary of the invention

[0003] In order to solve the above technical problems, a system and method for automatic parking mapping and positioning are provided.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] An automatic parking mapping and positioning system, comprising:

[0006] The image information receiving module receives continuous image information of the vehicle's surrounding environment captured by at least two monocular cameras.

[0007] Vehicle information assembly interface, receiving driving data information from the vehicle,

[0008] The processing unit generates feature points with depth information in each frame of the continuous image information of the vehicle's surrounding environment, and after matching the feature points with the vehicle's driving data information, constructs a three-dimensional point cloud map.

[0009] A storage module stores the three-dimensional point cloud map.

[0010] When the vehicle is repositioned, the processing unit can receive real-time image information of the vehicle's surrounding environment captured by the at least two monocular cameras in real time according to the image information receiving module, and use it to compare with the stored three-dimensional point cloud map to determine the positioning posture of the at least two monocular cameras.

[0011] The continuous image information of the vehicle's surrounding environment at least includes image information in front of the vehicle and behind the vehicle.

[0012] The continuous image information of the vehicle's surrounding environment includes at least image information in front of the vehicle, image information behind the vehicle, and image information on the left and right sides of the vehicle.

[0013] The processing unit has semantic operations and can remove dynamic feature points in the three-dimensional point cloud map.

[0014] The image information on the left and right sides of the vehicle has an image overlap area with the image information in front of the vehicle and the image information behind the vehicle.

[0015] as well as,

[0016] A method for automatic parking mapping and positioning, comprising:

[0017] Receive continuous image information of the vehicle's surrounding environment from at least two monocular cameras,

[0018] Receive driving data information from the vehicle,

[0019] Generate feature points with depth information in each frame of the continuous image information of the vehicle's surrounding environment, and after matching the feature points with the vehicle's driving data information, construct a three-dimensional point cloud map.

[0020] storing the constructed three-dimensional point cloud map,

[0021] When the vehicle is repositioned, the image information receiving module receives real-time image information of the vehicle's surrounding environment captured by the at least two monocular cameras in real time, and uses it to compare with the stored three-dimensional point cloud map to determine the positioning posture of the at least two monocular cameras.

[0022] The continuous image information of the vehicle's surrounding environment at least includes image information in front of the vehicle and behind the vehicle.

[0023] The continuous image information of the vehicle's surrounding environment includes at least image information in front of the vehicle, image information behind the vehicle, and image information on the left and right sides of the vehicle.

[0024] It also includes semantic operations, which can remove dynamic feature points in the three-dimensional point cloud map.

[0025] The image information on the left and right sides of the vehicle has an image overlap area with the image information in front of the vehicle and the image information behind the vehicle.

[0026] The present invention has the following advantages:

[0027] 1. When the automatic parking system establishes a parking lot map, since the system of the present invention receives continuous image information of the vehicle's surrounding environment captured by at least two monocular cameras, it can cover the field of view of all vehicle surrounding environments, that is, form a panoramic field of view, thereby improving the completeness of the three-dimensional point cloud map of the parking lot. At the same time, in order to achieve a more accurate three-dimensional point cloud map of the parking lot, the present invention can remove dynamic feature points in the three-dimensional point cloud map through semantic operations.

[0028] 2. Since the monocular cameras installed in the front, rear, left and right sides of the vehicle can replace the front and rear image frames, the distortion of the image edge caused by the lens can be reduced, thereby improving the completeness and accuracy of the three-dimensional point cloud map constructed in the parking lot area.

[0029] 3. When positioning automatic parking, such as when facing a parking lot with high repetitiveness or when the vehicle's travel angle is offset, due to the difference in the image information captured in real time by these monocular cameras, the three-dimensional point cloud map positioning posture of these monocular cameras corresponding to the parking lot can be quickly found, and then the vehicle positioning posture can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a structural schematic diagram of an automatic parking mapping and positioning system of the present invention;

[0031] Figure 2 It is a flow chart of a method for automatic parking mapping and positioning of the present invention;

[0032] Figure 3 is a schematic diagram of a monocular camera according to a first embodiment of the present invention being arranged on a vehicle;

[0033] Figure 4 It is a schematic diagram of a monocular camera according to a second embodiment of the present invention installed in a vehicle. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] In the description of the present invention, it is necessary to understand that the terms "upper" and "lower" etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0036] In the description of the present invention, “plurality” means two or more than two, unless otherwise clearly and specifically defined.

[0037] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two components or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0038] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0039] Please refer to Figure 1 The present invention discloses an automatic parking mapping and positioning system 100 including an image information receiving module 101 , a vehicle information assembly interface 102 , a processing unit 103 and a storage module 104 .

[0040] The image information receiving module 101 is used to receive continuous image information and has an image processing unit (ISP, Image Signal Processor), which can process functions such as lens correction, pixel correction, color interpolation, Bayer noise removal, white balance correction, color correction, gamma correction, color space conversion, etc. The image information receiving module 101 may generally have an LVDS (Low Voltage Differential Signaling) or MIPI CSI transmission interface (not marked). In the present invention, the continuous image information comes from at least two monocular cameras arranged on the vehicle, and the monocular camera is usually installed on the outside of the vehicle to avoid being blocked by the field of vision of the vehicle itself. It is mainly used to capture continuous image information of the surrounding environment of the vehicle. In order to obtain better image quality, the image information can be LVDS format information.

[0041] The vehicle information assembly interface 102 is connected to the vehicle CAN bus to receive driving data information from the vehicle. The driving data information of the vehicle includes, for example, vehicle speed, engine speed, steering angle, acceleration, gear position, etc., and may even include sensing information from an inertial measurement unit (IMU) mounted on the vehicle, a wheel speed meter, GPS, etc. that can measure the distance traveled by the vehicle.

[0042] The processing unit 103 is the main computing unit of the present invention, and the processor of the processing unit 103 is usually a DSP (digital signal processor). The DSP is suitable for performing various multiplication and addition operations (SOP: Sum of Products), such as: finite impulse response filter operation (FIR: Finite Impulse Response), infinite impulse response filter operation (IIR: Infinite Impulse Response), discrete Fourier transform (DFT: Discrete Fourier Transform), discrete cosine transform (DCT: Discrete Cosine Transform), dot product operation (Dot product), convolution operation (Convolution), and matrix polynomial evaluation operation, etc. The processing unit 103 is connected to the image information receiving module 101, and calculates the image information converted by the image information receiving module 101. The processing unit 103 is also connected to the vehicle information assembly interface 102 to receive driving data information from the vehicle. Therefore, according to the continuous image information and driving data information of the vehicle's surrounding environment, a three-dimensional point cloud map can be constructed when the automatic parking mapping function is turned on. When the vehicle is repositioned, the three-dimensional point cloud map is compared to obtain the positioning posture of at least two monocular cameras.

[0043] The main function of the storage module 104 is to store image information and / or various driving data information, as well as to store a three-dimensional point cloud map. The storage module 105 can be a built-in integrated circuit memory, or an external storage device, such as an SSD or SD card. At the same time, the storage module 104 is connected to the processing unit 103, and can store data according to the information generated by the processing unit 103.

[0044] Please refer to Figure 1 , Figure 2 and Figure 3 This is an embodiment of the operation of an automatic parking mapping and positioning system 100 according to the present invention. The automatic parking mapping and positioning system 100 of the present invention is installed on a vehicle and performs the following steps:

[0045] Step S01, receiving continuous image information of the vehicle's surrounding environment captured by at least two monocular cameras and receiving driving data information from the vehicle. When the user turns on the automatic parking function on the vehicle, the first mapping mode must be performed first. Therefore, after turning on the automatic parking function, the user must drive to a parking space (usually with a grid line drawn on it) or a specific parking space in the parking lot. During the user's self-driving process, the system 100 of the present invention receives the image information receiving module 101 installed in front of the vehicle and behind the vehicle (such as Figure 3 ) capture continuous image information of the vehicle's surroundings. These continuous image information involve internal and external parameters of the monocular cameras and must therefore be calibrated. In this embodiment, the monocular cameras have a wide viewing angle or a fisheye lens, so that the continuous image information captured by these monocular cameras installed in front of and behind the vehicle can cover the field of view of all the vehicle's surroundings, that is, to form a panoramic field of view. These monocular cameras must have a good focal length, and the shooting within a distance of 20 meters appears clear, so that the user can fully collect the scene images of the parking lot during self-driving, so that the vehicle can drive to a specific parking space or parking space. Each frame of the image information of the vehicle's surroundings has a corresponding complete three-dimensional point cloud map. At the same time, the vehicle information assembly interface 102 is connected to the vehicle CAN bus to receive driving data information from the vehicle. The driving data information of these vehicles is mainly the sensing information for measuring the vehicle's moving distance and speed, which is used as the actual distance information to match the continuous image information of the vehicle's surroundings.

[0046] Step S02, generating feature points with depth information in each frame of the continuous image information of the vehicle's surrounding environment. The processing unit 103 mainly extracts feature points from the continuous image information, and the feature point calculation adopted by the present invention is performed in a model-based manner instead of computer vision. It mainly estimates the descriptors of each feature point in each frame of the image information, especially the points, corners, and textures of common pillars, walls, road signs, symbols, etc. (but not limited to the above) in the parking lot area. As the vehicle moves, the descriptors of the feature points of the previous and next frames are paired one-to-one, and the projection changes of the feature points on the stereo coordinates after pairing are calculated, and finally the feature points with depth information in each frame of the continuous image information are obtained.

[0047] Step S03, matching the feature points with the driving data information of the vehicle. Since the image information is a record composed of continuous frames, the front and rear frames of the image reflect the time difference. In step S02, even if the processing unit 103 generates feature points with depth information for each frame of the continuous image information, there will be a certain difference between the actual distance traveled by the vehicle. Therefore, in this embodiment, the processing unit 103 receives the driving data information from the vehicle through the vehicle information assembly interface 102 according to step S01, especially the inertial measurement unit (Inertial Measurement Unit) and the wheel speed meter and other related sensing information that can measure the vehicle's moving distance. When the vehicle moving distance corresponding to the front and rear frames of the image is known, the feature points can be matched with the driving information of the vehicle according to the triangulation principle to obtain the depth information of the actual feature points. In addition, for the marking of all feature points in the image, the processing unit 103 has semantic operations, which can remove dynamic feature points in the three-dimensional point cloud map. The purpose is that for the first time the user builds the map, the parking area may include many dynamic objects, such as parked vehicles, pedestrians, movable fire-fighting equipment, etc. If these dynamic objects are regarded as feature points during the construction of the map and become part of the three-dimensional point cloud map, when the user subsequently turns on the automatic parking, the position of the dynamic objects changes or moves, causing the automatic parking function to fail in positioning. Therefore, through the semantic operation of the processing unit 103, the dynamic feature points in the area selected by the semantic operation can be deleted in each frame of the image information, that is, the information with dynamic feature points is not stored in the three-dimensional point cloud map. When the user executes the automatic parking function next time, the system can again add these deleted dynamic feature points to the area of ​​the three-dimensional point cloud map, and according to the new feature points generated by the image information captured at that time, appropriately update these areas that originally did not have feature points in the three-dimensional point cloud map, thereby improving the accuracy of the parking area map.

[0048] Step S04, storing the three-dimensional point cloud map. This step is to store the established three-dimensional point cloud map in the storage module 104, and the stored information may include the continuous image information captured by each monocular camera, and also include the relative coordinates of these monocular cameras corresponding to the three-dimensional point cloud map. The storage module 104 can even store the three-dimensional point cloud maps established by the user in different parking areas, or the three-dimensional point cloud maps of the same parking area at different times. Furthermore, it can store the mapping completeness index of each parking area after semantic calculation of dynamic feature points, as well as GPS information and inertial measurement unit information from the vehicle. The information stored in the storage module 104 can be read, rewritten, or transmitted to the cloud server through the network for the purpose of optimizing the map and exchanging information.

[0049] Step S05, compare the real-time image information of the vehicle surrounding environment captured by the monocular camera in real time with the stored three-dimensional point cloud map to determine the positioning posture of the at least two monocular cameras. When the user completes the establishment and storage of the three-dimensional point cloud map in step S04, when the automatic parking function is activated again and the vehicle is repositioned, the processing unit 103 reads the three-dimensional point cloud map stored in the storage unit 104, and simultaneously receives the real-time image information of the vehicle surrounding environment captured by these monocular cameras (in this embodiment, at least two monocular cameras are set in front of the vehicle and behind the vehicle) in real time. After calculating the feature points of these real-time image information and comparing them with the stored three-dimensional point cloud map at the same time, the positioning posture of these at least two monocular cameras can be quickly obtained, and then the coordinate position of the vehicle in the three-dimensional point cloud map is obtained, so as to achieve rapid positioning.

[0050] Since the at least two monocular cameras mounted on the front and rear of the vehicle in this embodiment can cover the field of view of the entire vehicle's surrounding environment, that is, form a panoramic field of view, the completeness of the three-dimensional point cloud map in the parking lot area is improved. In addition, when the vehicle is repositioned, the difference in the image information captured in real time by these monocular cameras can quickly find the positioning posture of these monocular cameras corresponding to the three-dimensional point cloud map of the parking lot area, and then obtain the vehicle positioning posture.

[0051] Please refer to Figure 4This is a second embodiment of the operation of an automatic parking mapping and positioning system 100 according to the present invention. The automatic parking mapping and positioning system 100 and the operation method of this embodiment are no different from the first embodiment, but the image information receiving module 101 in this embodiment receives the image information in front of the vehicle, the image information behind the vehicle, and the image information on the left and right sides of the vehicle, that is, the monocular camera is installed in front, behind, and on the left and right sides of the vehicle to capture continuous image information of the vehicle's surrounding environment to form a panoramic field of view. Since the viewing angle of a monocular camera is relatively wide, the credibility of the depth information of the relative image will be reduced. The lens carried by the monocular camera will easily cause distortion when the image is close to the edge of the image. The distorted image information will cause poor matching when constructing a three-dimensional point cloud map of the panoramic view of the parking lot, especially when matching the feature points of the front and rear frames of the image (step S03), and even abnormal matching of feature points on the same object. Therefore, in addition to receiving image information from the front and rear of the vehicle, this embodiment also receives image information from the left and right sides of the vehicle at the same time, and these image information from the left and right sides of the vehicle have image overlap areas with the image information from the front and rear of the vehicle. Since the image information captured by these monocular cameras is synchronized and real-time, in the image overlapping area, in addition to determining that the three-dimensional point cloud map of the panoramic view can be completely covered by the field of view of these monocular cameras, for these image overlapping areas, the processing unit 103 marks them without calculating the feature points of the image information. As the vehicle's moving posture changes, these overlapping and possibly highly distorted image information will be replaced by subsequent image information on the left and right sides of the vehicle. The replaced image information can reduce the distortion of the image edges caused by the lenses in the monocular cameras in front of and behind the vehicle, thereby greatly improving the accuracy of the three-dimensional point cloud map.

[0052] Similarly, since the monocular cameras installed in front of the vehicle, behind the vehicle, and on the left and right sides of the vehicle in this embodiment can cover the field of view of the entire vehicle's surrounding environment, that is, to form a panoramic field of view, and can reduce image distortion, thereby improving the completeness and accuracy of the three-dimensional point cloud map in the parking lot area. In addition, when the vehicle is repositioned, the difference in image information captured in real time by these monocular cameras can quickly find the positioning posture of these monocular cameras corresponding to the three-dimensional point cloud map of the parking lot area, and then obtain the vehicle positioning posture.

[0053] For users who use the automatic parking function, since the present invention includes at least two monocular cameras, when positioning the automatic parking, such as when facing a parking area with high repeatability, or when the vehicle is moving at an offset angle, due to the difference in real-time image information captured by these monocular cameras, the three-dimensional point cloud map positioning posture of these monocular cameras corresponding to the parking area can be quickly found, and then the vehicle positioning posture can be obtained. In order to achieve a more accurate three-dimensional point cloud map of the parking area, the present invention can remove dynamic feature points in the three-dimensional point cloud map through semantic operations. Therefore, by repeating the aforementioned steps S01 to S04, the accuracy of the parking area can be optimized.

[0054] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the methods and technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A system for automatic parking mapping and positioning, comprising: The image information receiving module receives continuous image information of the vehicle's surrounding environment captured by monocular cameras installed in front, rear, left and right sides of the vehicle, wherein the continuous image information of the vehicle's surrounding environment at least includes image information in front of the vehicle, image information in the rear of the vehicle, and image information on the left and right sides of the vehicle, and the image information on the left and right sides of the vehicle has an image overlap area with the image information in front of the vehicle and the image information in the rear of the vehicle. The vehicle information assembly interface receives driving data information from the vehicle, A processing unit generates feature points with depth information in each frame of the continuous image information of the vehicle's surrounding environment, and after matching the feature points with the vehicle's driving data information, constructs a three-dimensional point cloud map. A storage module stores the three-dimensional point cloud map. It is characterized in that when the vehicle is repositioned, the processing unit receives the real-time image information of the vehicle surrounding environment captured by the monocular camera in real time according to the image information receiving module, and uses it to compare with the stored three-dimensional point cloud map to determine the positioning posture of the monocular camera. The processing unit estimates the descriptor of each feature point, and as the vehicle moves, pairs the feature points one-to-one according to the descriptors of the feature points of the previous and next frames, and calculates the projection changes of the feature points on the stereo coordinates after pairing, thereby obtaining the feature points with depth information in each frame of the continuous image information of the vehicle's surrounding environment, and receives the driving data information from the vehicle through the vehicle information assembly interface to measure the moving distance of the vehicle, and matches the feature points with the driving data information of the vehicle based on the triangulation principle to obtain the depth information of the actual feature points. For the image overlap area, the processing unit gives a mark without calculating the feature points of the image information, and the image information in the image overlap area can be replaced by subsequent image information on the left and right sides of the vehicle as the vehicle's moving posture changes.

2. The automatic parking mapping and positioning system according to claim 1, characterized in that: The processing unit has semantic operations to remove dynamic feature points in the three-dimensional point cloud map.

3. A method for automatic parking mapping and positioning, comprising: receiving continuous image information of the vehicle's surrounding environment captured by monocular cameras installed in front, rear, and left and right sides of the vehicle, wherein the continuous image information of the vehicle's surrounding environment at least includes image information in front of the vehicle, image information in the rear of the vehicle, and image information on the left and right sides of the vehicle, and the image information on the left and right sides of the vehicle has an image overlap area with the image information in front of the vehicle and the image information in the rear of the vehicle, receiving driving data information from the vehicle, Generate feature points with depth information in each frame of the continuous image information of the vehicle's surrounding environment, and after matching the feature points with the vehicle's driving data information, construct a three-dimensional point cloud map. storing the constructed three-dimensional point cloud map, It is characterized in that when the vehicle is repositioned, the real-time image information of the vehicle surrounding environment captured by the monocular camera is received according to the image information receiving module, and is used to compare with the stored three-dimensional point cloud map to determine the positioning posture of the monocular camera. By estimating the descriptor of each feature point, as the vehicle moves, one-to-one pairing of the feature points is performed according to the descriptors of the feature points of the previous and next frames, and the projection change of the feature points on the stereo coordinates after pairing is calculated, the feature points with depth information in each frame of the continuous image information of the vehicle's surrounding environment are obtained, and by receiving the driving data information from the vehicle to measure the moving distance of the vehicle, and matching the feature points with the driving data information of the vehicle by the triangulation principle, the depth information of the actual feature points is obtained. The image overlap area is marked without calculating the feature points of the image information, and the image information in the image overlap area can be replaced by subsequent image information on the left and right sides of the vehicle as the moving position of the vehicle changes.

4. The method for automatic parking mapping and positioning as claimed in claim 3, characterized in that: It also includes semantic operations to remove dynamic feature points in the three-dimensional point cloud map.

Images