Parking positioning method, parking method and parking device based on image key frame

Abstract

The invention discloses a parking positioning method, a parking method and a parking device based on image key frames. The parking positioning method comprises the steps that S1, multi-source information around a parking vehicle and candidate parking space image data are collected and preprocessed; s2, registering and fusing the preprocessed multi-source information, and combining data with an image recognition result to obtain a data optimization set; and S3, identifying and extracting image key frames of candidate parking space image data in the data optimization set based on a deep learning model module. The invention belongs to the technical field of parking, and aims to solve the problem that large deviation exists between a parked vehicle and a parking space due to the fact that the parking space cannot be accurately identified, evaluated and positioned according to environmental perception data in the prior art. The technical effects are that accurate positioning of the vehicle and the candidate parking space can be realized conveniently, large deviation between the vehicle and the parking space after parking can be avoided, and the accuracy and convenience of parking can be improved.

Description

Technical Field

[0001] This application belongs to the technical field of parking, and specifically relates to a parking positioning method, a parking method, and a parking device based on image key frames. Background Art

[0002] With the rapid development of intelligent vehicle technology, the automatic parking system has become one of the important functions of modern vehicles and is even gradually regarded as a standard configuration of modern vehicles. The automatic parking system can complete the parking operation quickly and accurately, saving the driver's time for finding a parking space and reversing into the parking space. For novice drivers, the automatic parking system greatly reduces the difficulty and pressure of parking.

[0003] Currently, parking systems are mainly divided into semi-automatic parking systems and fully automatic parking systems. The semi-automatic parking system requires the driver to control the vehicle speed, while the system is responsible for the steering operation. This system is suitable for parallel parking, but cannot be used for perpendicular parking. The fully automatic parking system is not only responsible for the steering operation but also controls the acceleration and deceleration of the vehicle. The driver only needs to monitor the parking process.

[0004] However, most automatic parking systems mainly rely solely on sensors such as ultrasonic radars or lidar for environmental perception and cannot accurately identify, evaluate, and locate the parking space based on the environmental perception data, resulting in a large deviation between the parked vehicle and the parking space. Therefore, the present invention provides a parking positioning method, a parking method, and a parking device based on image key frames to solve the above problems. Summary of the Invention

[0005] This application provides a parking positioning method, a parking method, and a parking device based on image key frames, aiming to solve the problem that the prior art cannot accurately identify, evaluate, and locate the parking space based on the environmental perception data, resulting in a large deviation between the parked vehicle and the parking space.

[0006] In a first aspect, a parking positioning method, a parking method, and a parking device based on image key frames, the method includes:

[0007] S1: Collect multi-source information around the parking vehicle and image data of candidate parking spaces and preprocess them;

[0008] S2: Register and fuse the preprocessed multi-source information, combine the data with the image recognition results, and obtain an optimized data set;

[0009] S3: Based on the deep learning model module, identify and extract the image key frames of the candidate parking space image data in the optimized data set;

[0010] S4: The deep learning model module outputs the top corner coordinates of the bounding box of the candidate parking space according to the image key frames to represent the position and size of the parking space;

[0011] S5: Establish the corresponding positioning relationship between the candidate parking space and the actual physical space.

[0012] Optionally, the multi-source information includes the position information data and image data of the vehicle.

[0013] Optionally, the S2 specifically includes the following steps:

[0014] S2.1: Time synchronization: A unified clock source is formulated through a pulse generator to regularly correct the clocks of each sensor, eliminate the cumulative error of the clock source. At the same time, the equivalent information of non-synchronous sensors at the same moment is obtained through interpolation calculation;

[0015] S2.2: Spatial registration: All data of different sensors are converted into the same vehicle coordinate system through the mathematical model of interpolation and extrapolation method, ensuring the consistency of data from different sources in time and space;

[0016] S2.3: Information fusion: The data from different sensors are comprehensively processed through a fusion algorithm to obtain an optimized data set.

[0017] Optionally, the identification elements of the image key frame include the perspective and angle that comprehensively display the candidate parking space and timeliness.

[0018] Optionally, the S5 specifically includes the following steps:

[0019] S5.1: Convert the top corner coordinates of the bounding box in the image coordinate system into coordinates in the actual physical space;

[0020] S5.2: Establish the actual physical space coordinate system, and through coordinate transformation, establish the corresponding positioning relationship between the parking space in the key image frame and the actual physical space;

[0021] S5.3: In the actual physical space coordinate system, fuse and locate the converted top corner coordinates of the parking space bounding box with the current position information of the vehicle to determine the position relationship between the candidate parking space and the vehicle.

[0022] In the second aspect, a parking method based on the parking positioning method of the image key frame, the parking method includes:

[0023] Step 1: Obtain the multi-source information of the vehicle under the same clock source after image key frame registration and fusion from the optimized data set to obtain a data subset;

[0024] Step 2: Judge the feasibility of the candidate parking space and the current position and attitude of the vehicle according to the data subset;

[0025] Step 3: Determine the current position and attitude of the vehicle and plan a path from the current position to the parking space;

[0026] Step 4: Output the planned parking path, combine the coordinate information, and implement the automatic parking function.

[0027] Optionally, the current position of the vehicle in Step 2 includes whether the current heading of the vehicle is consistent with the direction of the candidate parking space.

[0028] In a third aspect, a parking device includes an acquisition module, a deep learning model module, a planning module, and an execution module. The acquisition module includes a vehicle sensor, a GPS positioning system, and an in-vehicle camera;

[0029] The basic structure of the deep learning model module is a convolutional neural network (CNN), which has strong feature extraction and classification capabilities;

[0030] The input end of the planning module is connected to the deep learning model module, and is used to output a specific parking path according to the current state information of the vehicle, the candidate parking space information, and the surrounding environment perception information;

[0031] The execution module is connected to the planning module, and is responsible for converting the instructions of the decision-making layer into the actual actions of the vehicle, and can receive the parking information sent by the planning module, so as to facilitate the subsequent execution of the parking operation.

[0032] Compared with the prior art, the present application has at least the following beneficial effects:

[0033] Based on the further analysis and research of the problems in the prior art, the present invention makes the key frames of the candidate parking space image and the parking vehicle position coordinates fused and unified by identifying and converting the key frames of the candidate parking space image, highlighting the relationship between the top corner coordinates of the candidate parking space bounding box and the parking vehicle position, so as to facilitate the accurate positioning of the vehicle and the candidate parking space, avoid large deviations between the vehicle and the parking space after parking, and thus improve the accuracy and convenience of parking. Description of the Drawings

[0034] Figure 1 It is a schematic flowchart of a parking positioning method, a parking method, and a parking device based on image key frames provided by an embodiment of the present application;

[0035] Figure 2 It is a schematic module diagram of a parking positioning method, a parking method, and a parking device based on image key frames provided by an embodiment of the present application; Detailed Embodiments

[0036] To make the objectives, technical solutions and advantages of this application more clear and understandable, the following further elaborates on this application in conjunction with the accompanying drawings and embodiments.

[0037] As Figure 1 shown, the parking position determination method based on image key frames provided by this application includes the following steps:

[0038] S1: Collect multi-source information around the parking vehicle and candidate parking space image data and perform preprocessing; the multi-source information includes vehicle position information data and image data.

[0039] The preprocessing steps include denoising, enhancement, and contrast adjustment. The denoising operation uses methods such as Gaussian filtering to eliminate noise and interference in the image. The enhancement operation uses the method of histogram equalization to improve the contrast and clarity of the image. The contrast adjustment operation of the candidate parking space image data can be adjusted according to the actual situation through an image processing tool (such as Adobe Photoshop) to ensure that the contrast between the parking space and the surrounding environment is high enough for subsequent recognition and positioning.

[0040] The position information includes the GPS positioning position of the vehicle, the relative distance between the vehicle and surrounding objects, the relative distance angle between the vehicle and surrounding objects, and the visual image around the vehicle.

[0041] S2: Register and fuse the preprocessed multi-source information, combine the data with the image recognition results, and obtain an optimized data set to improve the accuracy of parking space positioning. Since the operating frequencies, data formats, and coordinate systems of different sensors may be different, it is necessary to perform time synchronization and spatial registration on them.

[0042] S2 specifically includes the following steps:

[0043] S2.1: Time synchronization: A unified clock source is established through a pulse generator to regularly correct the clocks of each sensor, eliminate the cumulative error of the clock source, and at the same time, obtain the equivalent information of non-synchronized sensors at the same moment through interpolation calculation. Even if the clock sources are the same, the acquisition moments of different sensors may still be different. For example, the acquisition frequencies of in-vehicle cameras and vehicle sensors may be different, resulting in mismatched data acquisition time points.

[0044] S2.2: Spatial registration: All data from different sensors are converted into the same vehicle coordinate system through a mathematical model of interpolation and extrapolation, ensuring the consistency of different data in time and space, thus providing accurate and comprehensive environmental perception information for automatic parking.

[0045] For example, during automatic parking, the data and images of vehicle sensors such as in-vehicle cameras, GPS positioning, and IMU are all converted into the vehicle coordinate system for subsequent data processing and analysis.

[0046] S2.3: Information fusion: The data from different sensors are comprehensively processed through a fusion algorithm to obtain an optimized data set. The fusion algorithm can utilize the redundancy and complementarity of the data to improve the accuracy and reliability of the information.

[0047] For example, the precise ranging ability of lidar and the visual recognition ability of the camera can be combined to jointly identify the position and size of the parking space, so that the system can accurately judge the position and relative speed of obstacles in the parking space, issue an alarm in a timely manner or take emergency braking measures to improve parking safety.

[0048] S3: Based on the deep learning model module, identify and extract the image key frames of the candidate parking space image data in the optimized data set. The identification elements of the image key frames need to comprehensively display the perspective and angle of the candidate parking space, so as to be able to comprehensively display the overall shape of the parking space and the surrounding environment, which helps the model better understand the spatial position of the parking space and the relative relationship with other objects; at the same time, it needs to be time-sensitive and be able to represent the latest frame of the current parking space state to ensure the real-time and accuracy of the information.

[0049] S4: The deep learning model module outputs the top corner coordinates of the bounding box of the candidate parking space to represent the position and size of the parking space. The top corner coordinates of the bounding box include the upper left coordinate, the upper right coordinate, the lower left coordinate, and the lower right coordinate.

[0050] S5: Establish the corresponding positioning relationship between the candidate parking space and the actual physical space.

[0051] S5 specifically includes the following steps:

[0052] S5.1: Convert the top corner coordinates of the bounding box in the image coordinate system into the coordinates in the actual physical space. During the conversion process, factors such as the internal and external parameters of the camera and image distortion correction need to be considered to ensure the accuracy of the coordinate conversion.

[0053] S5.2: Establish the actual physical space coordinate system, and through coordinate conversion, establish the corresponding positioning relationship between the parking space in the key image frame and the actual physical space.

[0054] A three-dimensional coordinate system with the vehicle center or a specific point as the origin in the actual physical space, the vehicle driving direction as the X-axis, perpendicular to the driving direction and pointing to the left side of the vehicle as the Y-axis, and perpendicular to the ground upward as the Z-axis. Through the inverse process of perspective projection, the vertex coordinates of the bounding box in the image coordinate system are converted into three-dimensional coordinates in the camera coordinate system; then, using the external camera parameters (rotation matrix and translation vector), the coordinates in the camera coordinate system are converted into coordinates in the vehicle body coordinate system to complete the coordinate conversion.

[0055] S5.3: In the actual physical space coordinate system, the converted vertex coordinates of the parking space bounding box are fused with the current position information of the vehicle to determine the positional relationship between the candidate parking space and the vehicle.

[0056] The present invention makes the key image frames of the candidate parking space fused and unified with the position coordinates of the parking vehicle by identifying and converting the key image frames of the candidate parking space, highlighting the positional relationship between the vertex coordinates of the candidate parking space bounding box and the parking vehicle position, so as to facilitate the precise positioning of the vehicle and the candidate parking space, avoid large deviations between the vehicle and the parking space after parking, and thus improve the accuracy and convenience of parking.

[0057] In one embodiment, a parking method based on key image frames is also provided; specifically, it includes the following steps:

[0058] Step 1: Obtain the multi-source information of the vehicle under the same clock source after image key frame registration and fusion from the data optimization collection to obtain a data subset.

[0059] Step 2: Judge the feasibility of the candidate parking space and the current position and attitude of the vehicle according to the data subset; it is necessary to judge whether the current heading of the vehicle is consistent with the direction of the candidate parking space. If there is a large deviation between the vehicle heading and the parking space direction and adjustment may be required to ensure smooth parking, then the candidate parking space is considered feasible.

[0060] If the judgment result is infeasible, it is necessary to find other candidate parking spaces or adjust the vehicle attitude again, and repeat the above steps for analysis.

[0061] Step 3: Determine the current position and attitude of the vehicle and plan a path from the current position to the parking space.

[0062] Step 4: Output the planned parking path and combine the coordinate information to realize the automatic parking function.

[0063] As Figure 2 shown, in one embodiment, a parking device based on key image frames is also provided, including an acquisition module, a deep learning model module, a planning module, and an execution module to implement the above-mentioned parking positioning method and parking method based on key image frames.

[0064] The acquisition module includes vehicle sensors, a GPS positioning system, and an on-vehicle camera. The vehicle sensors are used to measure the relative distance and angle between the vehicle and surrounding objects. The GPS positioning system obtains the absolute position information of the vehicle. The on-vehicle camera captures visual information around the vehicle, such as lane lines and parking space markings, providing a more comprehensive environmental perception around the vehicle.

[0065] The basic structure of the deep learning model module is a convolutional neural network (CNN), which has strong feature extraction and classification capabilities. During the extraction process, the backpropagation algorithm can be used to optimize and adjust the model to improve the recognition accuracy and positioning precision of the model. At the same time, data augmentation techniques can be used to expand the dataset and improve the generalization ability of the model.

[0066] The input end of the planning module is connected to the deep learning model module and is used to output a specific parking path based on the current vehicle state information, candidate parking space information, and surrounding environmental perception information. The planning module includes an environmental modeling module and the Dijkstra algorithm.

[0067] The purpose of environmental modeling is to provide a simple and efficient environmental representation for subsequent path planning and navigation. Based on the environmental model, the parking path planning module needs to use the Dijkstra algorithm to perform path retrieval to find an optimal path from the starting point to the ending point. The initial path found needs to be optimized and smoothed. This usually involves interpolating, fitting, or adjusting the path to ensure it meets the driving requirements and safety requirements of the vehicle.

[0068] The environmental perception information is mainly provided by the acquisition module.

[0069] The execution module is connected to the planning module and is responsible for converting the instructions at the decision-making level into actual actions of the vehicle. It can receive the parking information sent by the planning module for subsequent parking operations. At the same time, the execution module includes the dynamic model of the vehicle and can control actions such as the steering, acceleration, and braking of the vehicle.

[0070] Among them, the algorithm model of the interpolation and extrapolation method in S2.2 generally refers to the optimization algorithm model for spatial registration, not limited to the interpolation and extrapolation method. For example, it can be the least squares algorithm model; mainly define the fitness function through the algorithm model to evaluate the registration effect, and use the iterative optimization method to find the best registration parameters, and describe the motion state of the target, so as to more easily achieve the fusion and registration of multi-sensor data.

[0071] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

Claims

1. A parking positioning method based on image key frames, characterized in that: The method comprises: S1: Collect and pre-process the multi-source information around the parking vehicle and the image data of the candidate parking spaces; S2: register and fuse the preprocessed multi-source information, combine the data with the image recognition results, and obtain the optimized data collection; S3: identifying and extracting image key frames of candidate parking space image data in the data optimization collection based on the deep learning model module; S4: The deep learning model module outputs the coordinates of the top corners of the bounding box of the candidate parking space according to the image key frame to indicate the location and size of the parking space; S5: Establishing a corresponding positioning relationship between the candidate parking spaces and the actual physical space.

2. The parking positioning method based on image key frames according to claim 1, characterized in that: The multi-source information includes location information data and image data of the vehicle.

3. The parking positioning method based on image key frames according to claim 1, characterized in that: The S2 specifically includes the following steps: S2.1: Time synchronization: A unified clock source is established through a pulse generator, and the clocks of each sensor are regularly corrected to eliminate the accumulated error of the clock source. At the same time, equivalent information of asynchronous sensors at the same time is obtained through interpolation calculation; S2.2: Spatial registration: All data from different sensors are converted into the same vehicle coordinate system through a mathematical model of interpolation and extrapolation, ensuring the consistency of data from different sensors in time and space; S2.3: Information fusion: The data from different sensors are processed comprehensively through fusion algorithms to obtain an optimized data collection.

4. The parking positioning method based on image key frames according to claim 1, characterized in that: The identification elements of the image key frame include the viewing angle and angle of the candidate parking space being fully displayed and being timely.

5. The parking positioning method based on image key frames according to claim 1, characterized in that: The S5 specifically includes the following steps: S5.1: Convert the coordinates of the top corners of the bounding box in the image coordinate system to the coordinates in the actual physical space; S5.2: Establishing an actual physical space coordinate system, and establishing a corresponding positioning relationship between the parking space in the key image frame and the actual physical space through coordinate transformation; S5.3: In the actual physical space coordinate system, the converted parking space boundary box vertex coordinates are integrated with the current position information of the vehicle to determine the positional relationship between the candidate parking space and the vehicle.

6. A parking method based on the parking positioning method based on image key frames according to any one of claims 1 to 5, characterized in that: The parking method comprises: Step 1: Obtain multi-source information of vehicles under the same clock source after image key frame registration and fusion from the data optimization collection to obtain a data subset; Step 2: Determine the feasibility of the candidate parking space and the current position and posture of the vehicle based on the data subset; Step 3: Determine the current position and posture of the vehicle and plan a path from the current position to the parking space; Step 4: Output the planned parking path and combine it with the coordinate information to realize the automatic parking function.

7. The parking method according to claim 6, characterized in that: The current position of the vehicle in step 2 includes whether the current heading of the vehicle is consistent with the direction of the candidate parking space.

8. A parking device, comprising a collection module, a deep learning model module, a planning module, and an execution module, characterized in that: The acquisition module includes vehicle sensors, GPS positioning system and vehicle-mounted cameras; The basic structure of the deep learning model module is a convolutional neural network (CNN), which has strong feature extraction and classification capabilities; The input end of the planning module is connected to the deep learning model module, and is used to output a specific parking path according to the vehicle's current state information, candidate parking space information, and surrounding environment perception information; The execution module is connected to the planning module, and is responsible for converting the instructions of the decision-making layer into actual actions of the vehicle. It can receive the parking information sent by the planning module to facilitate the subsequent execution of parking operations.

Images