Parking space positioning method and system based on multi-sensor fusion

Abstract

The application discloses a parking space positioning method and system based on multi-sensor fusion. The method comprises the following steps: performing feature matching on the real scene image features at the current time and the real scene image features at the previous time; if the matching is successful, calculating the image acquisition module pose at the current time; if the matching fails, calculating the image acquisition module pose at the current time according to the relative position relationship between the inertial measurement module and the image acquisition module and the inertial measurement information at the previous time, and taking the image acquisition module pose at the current time as the patrol vehicle pose; associating and binding the real scene image features, the patrol vehicle pose and the parking space positioning information, taking the patrol vehicle pose at the first frame time as the initial pose, and generating a parking space map; obtaining the to-be-tested patrol vehicle pose according to the to-be-tested real scene image features, comparing the to-be-tested patrol vehicle pose with all the patrol vehicle poses in the parking space map, obtaining the patrol vehicle pose closest to the to-be-tested patrol vehicle pose, inquiring the corresponding parking space positioning information, and taking the parking space positioning information as the parking space positioning information corresponding to the to-be-tested real scene image.

Description

Technical Field

[0001] This invention relates to the field of smart parking technology, and in particular to a parking space positioning method and system based on multi-sensor fusion. Background Technology

[0002] Parking space location information is a crucial component of smart city construction and urban traffic management. Parking space location technology plays a vital role in scenarios such as urban static traffic management, parking fee management, and comprehensive urban governance. The primary objective of parking space location technology is to obtain the location information of parking spaces.

[0003] Berth positioning methods based on satellite positioning technology rely on geographic latitude and longitude information to locate berths. However, satellite positioning information is easily affected by factors such as weather, obstruction, and base station service quality, which can lead to positioning errors or failures. Summary of the Invention

[0004] The purpose of this invention is to address the technical problem in berth positioning methods based on satellite positioning technology, where satellite positioning information is easily affected by factors such as weather, obstruction, and base station service quality, leading to positioning errors or failures. To achieve the above objective, this invention provides a berth positioning method and system based on multi-sensor fusion.

[0005] This invention provides a berth positioning method based on multi-sensor fusion, comprising:

[0006] Acquire real-world images and inertial measurement information;

[0007] Feature extraction is performed on each frame of the real-scene image to obtain real-scene image features, and feature matching is performed between the real-scene image features at the current moment and the real-scene image features at the previous moment.

[0008] If the match is successful, the pose of the image acquisition module at the current moment is calculated based on the real-scene image features at the current moment and the real-scene image features at the previous moment.

[0009] If the matching fails, the relative positional relationship between the inertial measurement module and the image acquisition module is obtained, and the image acquisition module pose at the current moment is calculated based on the relative positional relationship and the inertial measurement information at the previous moment. The image acquisition module pose at the current moment is the inspection vehicle pose at the current moment.

[0010] Obtain the berth positioning information at the current moment, associate and bind the real scene image features, the patrol vehicle pose, and the berth positioning information at the current moment, and establish the starting point with the patrol vehicle pose at the first frame as the initial pose to generate a berth map.

[0011] Acquire a real-world image to be tested, extract features from the real-world image to obtain the features of the real-world image to be tested, and obtain the position pose of the inspection vehicle to be tested based on the features of the real-world image to be tested, and compare the position pose of the inspection vehicle to be tested with all the position poses of the inspection vehicles in the parking space map to obtain the position pose of the inspection vehicle closest to it.

[0012] The location information of the nearest inspection vehicle position is queried in the berth map and used as the location information of the berth corresponding to the real scene image to be tested.

[0013] The image acquisition module is used to acquire the real-scene image, and the inertial measurement module is used to measure the inertial measurement information.

[0014] In one embodiment, the image acquisition module acquires real-scene images and performs feature extraction on each frame of the real-scene image. Before obtaining the real-scene image features, the method further includes:

[0015] Determine if a historical berth map exists;

[0016] If the historical berth map exists, then obtain the berth location information based on the historical berth map;

[0017] If the historical berth map does not exist, the image acquisition module is executed to acquire real-scene images, and feature extraction is performed on each frame of the real-scene image to obtain the real-scene image features.

[0018] In one embodiment, obtaining the parking space positioning information at the current moment, associating and binding the real-scene image features, the patrol vehicle pose, and the parking space positioning information at the current moment, and establishing a starting point with the patrol vehicle pose at the first frame as the initial pose to generate a parking space map, includes:

[0019] The berth image is detected and identified to obtain the berth number, and the berth location information is obtained based on the berth number;

[0020] Alternatively, obtain the location information of the inspection vehicle, compare the location information of the inspection vehicle with the location information list, and take the location information that is closest to the location information of the inspection vehicle as the parking space location information;

[0021] Alternatively, the berth location information can be obtained based on the radio frequency identification tag.

[0022] In one embodiment, after the image acquisition module acquires real-scene images and extracts features from each frame of the real-scene image to obtain real-scene image features, before performing feature matching between the real-scene image features at the current moment and the real-scene image features at the previous moment, the method further includes:

[0023] Determine whether the real-scene image at the current moment is the first frame image;

[0024] If so, continue to acquire the real-scene images to obtain multiple frames of the real-scene images;

[0025] If not, then perform feature matching on the real-scene image features of two adjacent real-scene images.

[0026] In one embodiment, if the historical berth map exists, obtaining berth location information based on the historical berth map includes:

[0027] If the historical berth map exists, then the current new real-scene image is obtained, and features are extracted from the current new real-scene image to obtain the features of the current new real-scene image;

[0028] Search the historical berth map for matching historical real-scene image features that match the current new real-scene image features;

[0029] Based on the features of the matched historical real-world images, the corresponding matched historical inspection vehicle pose is obtained, and the matched historical inspection vehicle pose is used as the initial pose.

[0030] Using the initial pose as the starting point, the historical berth map is updated according to the map update method to obtain an updated berth map, and the berth positioning information is obtained based on the updated berth map.

[0031] In one embodiment, if the historical berth map exists, obtaining berth location information based on the historical berth map includes:

[0032] If the historical berth map exists, the current new real-scene image is obtained, and each feature point in the current new real-scene image is extracted to form a current new feature point set, which includes multiple current new feature points;

[0033] The number of historical real-world images containing the current new feature point is found in the historical berth map and used as the first observation frame number;

[0034] Find all historical real-view images adjacent to the current new real-view image in the historical berth map to form an adjacent historical real-view image set;

[0035] Determine whether there exists a historical feature point in the adjacent historical real-scene image set that matches the current new feature point;

[0036] If it exists, then find the number of adjacent historical real-scene images that match the current new feature point in the adjacent historical real-scene image set, and use it as the second observation frame number;

[0037] The first observation frame number is compared with the second observation frame number. The real-scene image corresponding to the maximum observation frame number is retained. The historical berth map is updated according to the feature points of the real-scene image corresponding to the maximum observation frame number to obtain the updated berth map.

[0038] If it does not exist, the current new feature point is added to the historical berth map for updating, and the updated berth map is obtained.

[0039] In one embodiment, after obtaining the berth positioning information by querying the nearest pose information in the berth map, the method further includes:

[0040] Send a stop data acquisition command to the image acquisition module and the inertial measurement module;

[0041] Store the berth map.

[0042] In one embodiment, the present invention provides a berth positioning system based on multi-sensor fusion, comprising:

[0043] The data receiving module is used to acquire real-scene images and inertial measurement information;

[0044] The feature matching module is used to extract features from each frame of the real-scene image, obtain real-scene image features, and perform feature matching between the real-scene image features at the current moment and the real-scene image features at the previous moment.

[0045] The image pose calculation module is used to calculate the pose of the image acquisition module at the current moment based on the real scene image features at the current moment and the real scene image features at the previous moment if the match is successful.

[0046] An inertial measurement module is used to obtain the relative positional relationship between the inertial measurement module and the image acquisition module if the matching fails, and to calculate the image acquisition module pose at the current moment based on the relative positional relationship and the inertial measurement information at the previous moment, wherein the image acquisition module pose at the current moment is the inspection vehicle pose at the current moment.

[0047] The berth map generation module is used to obtain the berth positioning information at the current moment, associate and bind the real scene image features, the patrol vehicle position pose, and the berth positioning information at the current moment, and establish the starting point with the patrol vehicle position pose at the first frame moment to generate the berth map.

[0048] The pose query module is used to extract features from the real-scene image to be tested, obtain the features of the real-scene image to be tested, obtain the pose of the inspection vehicle to be tested based on the features of the real-scene image to be tested, and compare the pose of the inspection vehicle to be tested with all the poses of the inspection vehicles in the parking space map to obtain the pose of the nearest inspection vehicle.

[0049] The berth positioning module is used to query the berth positioning information corresponding to the nearest inspection vehicle position in the berth map, and use it as the berth positioning information corresponding to the real scene image to be tested.

[0050] The image acquisition module is used to acquire the real-scene image, and the inertial measurement module is used to measure the inertial measurement information.

[0051] In one embodiment, the system further includes:

[0052] The map has a judgment module used to determine whether a historical berth map exists;

[0053] The first module for determining the existence of a map is used to obtain berth location information based on the historical berth map if the historical berth map exists.

[0054] The second map existence judgment module is used to execute the image acquisition module to acquire real-scene images if the historical berth map does not exist.

[0055] In one embodiment, the berth map generation module includes:

[0056] The berth location information acquisition module 1 is used to detect and identify berth images, obtain berth numbers, and obtain berth location information based on the berth numbers;

[0057] Alternatively, the second berth positioning information acquisition module is used to acquire the patrol vehicle positioning information, compare the patrol vehicle positioning information with the positioning information list, and take the positioning information closest to the patrol vehicle positioning information as the berth positioning information;

[0058] Alternatively, the berth location information acquisition module three is used to acquire the berth location information based on the radio frequency identification tag.

[0059] In one embodiment, the system further includes:

[0060] The first frame image determination module is used to determine whether the real-scene image at the current moment is the first frame image;

[0061] Image judgment result module one is used to continue acquiring the real scene image if the result is positive, thereby obtaining multiple frames of the real scene image;

[0062] The second image judgment result module is used to perform feature extraction on each frame of the real scene image if the condition is not met, obtain the real scene image features, and perform feature matching between the real scene image features at the current moment and the real scene image features at the previous moment.

[0063] In one embodiment, the map existence determination module one includes:

[0064] The current new real-scene image feature acquisition module is used to acquire the current new real-scene image if the historical berth map exists, and to extract features from the current new real-scene image to obtain the current new real-scene image features.

[0065] The historical real-scene image feature acquisition module is used to query the historical berth map for matching historical real-scene image features that match the current new real-scene image features;

[0066] The initial pose acquisition module is used to obtain the corresponding historical patrol vehicle pose based on the features of the historical matching images, and to use the historical matching patrol vehicle pose as the initial pose.

[0067] The berth map update module 1 is used to update the historical berth map according to the map update method, starting from the initial pose, to obtain an updated berth map, and to obtain the berth positioning information based on the updated berth map.

[0068] In one embodiment, the map existence determination module one includes:

[0069] The current new feature point set acquisition module is used to acquire the current new real-scene image if the historical berth map exists, and extract each feature point in the current new real-scene image to form a current new feature point set, which includes multiple current new feature points;

[0070] The first observation frame acquisition module is used to find the number of historical real-world images containing the current new feature point in the historical berth map, and use it as the first observation frame number.

[0071] The adjacent historical real-scene image set acquisition module is used to find all historical real-scene images adjacent to the current new real-scene image in the historical berth map, and form an adjacent historical real-scene image set;

[0072] The adjacent historical set feature matching judgment module is used to determine whether there is a historical feature point in the adjacent historical real scene image set that matches the current new feature point;

[0073] The second observation frame acquisition module is used to find, if it exists, the number of adjacent historical real-scene images that match the current new feature point in the adjacent historical real-scene image set, and use it as the second observation frame number.

[0074] The second module for updating the berth map is used to compare the first observation frame number with the second observation frame number, retain the real-scene image corresponding to the maximum observation frame number, and update the historical berth map according to the feature points of the real-scene image corresponding to the maximum observation frame number to obtain the updated berth map.

[0075] The third module for updating the berth map is used to add the current new feature point to the historical berth map for updating if it does not exist, thereby obtaining the updated berth map.

[0076] In one embodiment, the system further includes:

[0077] The instruction sending module is used to send a stop data acquisition instruction to the image acquisition module and the inertial measurement module;

[0078] A storage module is used to store the berth map.

[0079] The aforementioned berth positioning method and system based on multi-sensor fusion calculates the pose of the inspection vehicle through the collaborative calculation of feature information from real-scene images and inertial measurement information. It then associates and binds the real-scene image features, the inspection vehicle pose, and the berth positioning information to form a berth map. This berth map establishes a correspondence between berth positioning information and surrounding real-scene image information. When berth positioning is needed, the berth positioning information can be obtained by calculating the inspection vehicle pose from the real-scene image and querying the berth map, thus achieving berth positioning. The berth positioning method based on multi-sensor fusion provided by this invention describes the pose change process from different perception dimensions, incorporating relevant feature information from multiple directions related to the berth. This ensures the robustness and continuity of the measurement results, making the measurement values ​​more detailed and accurate, resulting in higher stability and better robustness in berth positioning. This avoids the positioning errors or failures caused by weather conditions associated with single-satellite positioning. Attached Figure Description

[0080] Figure 1 This is a flowchart illustrating the steps of the berth positioning method based on multi-sensor fusion provided by the present invention.

[0081] Figure 2This is a schematic diagram of the berth positioning system based on multi-sensor fusion provided by the present invention. Detailed Implementation

[0082] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0083] Please see Figure 1 This invention provides a berth positioning method based on multi-sensor fusion, comprising:

[0084] S10, acquire real-scene images and inertial measurement information;

[0085] S20: Extract features from each frame of real-world image, obtain real-world image features, and perform feature matching between the real-world image features at the current moment and the real-world image features at the previous moment.

[0086] S30, if the match is successful, calculate the pose of the image acquisition module at the current moment based on the real scene image features at the current moment and the real scene image features at the previous moment.

[0087] S40, if the matching fails, the relative position relationship between the inertial measurement module and the image acquisition module is obtained, and the image acquisition module pose at the current moment is calculated based on the relative position relationship and the inertial measurement information at the previous moment. The image acquisition module pose at the current moment is the inspection vehicle pose at the current moment.

[0088] S50: Obtain the berth positioning information at the current moment, associate and bind the real scene image features, inspection vehicle position pose, and berth positioning information at the current moment, and establish the starting point with the inspection vehicle position pose at the first frame as the initial pose to generate a berth map.

[0089] S60, acquire the real-scene image to be tested, extract features from the real-scene image to be tested, acquire the features of the real-scene image to be tested, and acquire the position pose of the inspection vehicle to be tested based on the features of the real-scene image to be tested, and compare the position pose of the inspection vehicle to be tested with the position poses of all inspection vehicles in the parking space map to obtain the position pose of the nearest inspection vehicle.

[0090] S70 queries the parking space location information corresponding to the nearest inspection vehicle position in the parking space map, and uses it as the parking space location information corresponding to the real scene image to be tested.

[0091] The image acquisition module is used to acquire real-world images, and the inertial measurement module is used to measure inertial measurement information.

[0092] In this embodiment, the image acquisition module and the inertial measurement module are mounted on the inspection vehicle. The image acquisition module can be a device capable of image acquisition. The inertial measurement module can be a device capable of inertial measurement data acquisition. Using an image sensor (which can also be understood as a camera), surrounding real-world image information is acquired, and feature analysis is performed on each real-world image to extract features and obtain real-world image feature information. In one embodiment, multi-level pyramid resampling is performed on the real-world images, and image feature information is analyzed at different pyramid layers.

[0093] If the current real-world image is not the first frame captured by the image acquisition module, it can be understood that the image acquisition module has captured at least two frames of real-world images, thus enabling feature matching based on two adjacent real-world images.

[0094] For the current frame real-scene image I a Compared to the previous real-world image I b Feature extraction is performed to obtain feature point sets F. a ={P i |i∈[1,N]},F b ={P j |j∈[1,M]}. The feature point set F a With F b Matching is performed. When the number of matching points is greater than a preset threshold, it means that the feature matching of the two real-scene images is successful. Step S30 is executed to reconstruct the image using the feature points in the two adjacent real-scene images. After successful reconstruction, the transformation relationship between the two adjacent real-scene images is calculated to obtain the pose of the image acquisition module at the current moment.

[0095] If feature matching fails, step S40 is executed. Since feature matching between two adjacent real-world images fails, pose calculation cannot be performed using the features of the current real-world image and the features of the previous real-world image. Therefore, the inertial measurement module (IMT) is used for pose calculation. The IMT module acquires inertial measurement information from the previous moment, including position and velocity information. The position and velocity information measured by the IMT module can represent the position and velocity information of the inspection vehicle. Based on the motion inertial model, the spatial displacement, velocity, and rotation information of the inspection vehicle at the current moment can be calculated based on the inertial measurement information from the previous moment. Based on the pre-calibrated relative positional relationship between the IMT module and the image acquisition module, and the inertial measurement information, the pose of the image acquisition module at the current moment is calculated. The pose of the image acquisition module at the current moment can represent the pose of the inspection vehicle at the current moment, thus obtaining the pose information of the inspection vehicle. The pose of the inspection vehicle can be understood as the position and attitude of the inspection vehicle, or as the position and attitude of the image acquisition module mounted on the inspection vehicle.

[0096] The berth map is the link for querying berth information through pose. The berth map includes real-scene image features, patrol vehicle pose information, and berth location information, all of which are interconnected and bound together. Each real-scene image feature constitutes a map point in the berth map, which is mutually bound to the patrol vehicle pose information and berth location information to form a complete berth map. If no available berth map is detected, a berth map is built starting from the patrol vehicle pose at the first frame. Based on the established berth map, it can be continuously updated. Through the established berth map, other relevant information can be obtained using the current real-scene image features, patrol vehicle pose information, or berth location information to achieve real-time berth location. Berth location information includes all information related to the berth, such as berth number, berth latitude and longitude, municipal information, etc.

[0097] The real-world image to be tested can be understood as the current real-world image acquired by the image acquisition module, used to achieve the location detection of the berth. Feature extraction is performed on the real-world image to be tested, and the extraction process is the same as the feature extraction step in S10, obtaining the features of the real-world image to be tested. The process of obtaining the pose information of the inspection vehicle to be tested based on the features of the real-world image to be tested is the same as the calculation process in S30, obtaining the corresponding pose of the inspection vehicle to be tested. A search is performed on the generated berth map to obtain the pose of the inspection vehicle closest to the location of the inspection vehicle to be tested. Further, in the berth map, the berth positioning information corresponding to the pose of the inspection vehicle closest to the location is found to obtain the berth positioning corresponding to the real-world image to be tested.

[0098] The berth positioning method based on multi-sensor fusion provided by this invention calculates the pose of an inspection vehicle by collaboratively using feature information from real-scene images and inertial measurement information. It then associates and binds the real-scene image features, the inspection vehicle pose, and the berth positioning information to form a berth map. This berth map establishes a correspondence between berth positioning information and surrounding real-scene image information. When berth positioning is needed, the berth positioning information can be obtained by calculating the inspection vehicle pose from the real-scene image and querying the berth map, thus achieving berth positioning. This multi-sensor fusion-based berth positioning method describes the pose change process from different perception dimensions, incorporating relevant feature information from multiple directions related to the berth. This ensures the robustness and continuity of the measurement results, making the measurement values ​​more detailed and accurate, resulting in higher stability and better robustness in berth positioning. This avoids the positioning errors or failures caused by weather conditions associated with single-satellite positioning.

[0099] In one embodiment, the berth map also includes detailed location information such as berth number and road administration data, increasing the diversity of berth information on the map and providing rich data for berth location. Therefore, when a patrol vehicle searches for a berth on the map using its position and orientation, it can obtain various location data about the berth, and then obtain more accurate berth location information based on the confidence weights of the location data, improving the success rate and accuracy of berth location.

[0100] In one embodiment, S10, before the image acquisition module acquires the real-scene image, the method further includes:

[0101] S01, Determine if a historical berth map exists;

[0102] S02, If a historical berth map exists, obtain the berth location information based on the historical berth map;

[0103] S03, if no historical berth map exists, execute the image acquisition module to acquire real-world images.

[0104] In this embodiment, before executing S10, it can be determined whether a historical berth map exists, or in other words, whether a usable historical berth map exists. A historical berth map is a berth map already established in the inspection vehicle's system. If a historical berth map exists, its information can be read, and the corresponding berth location information can be searched based on the historical berth map. If no historical berth map exists, a new berth map can be created. When creating a new berth map, S10 can be executed to open the image acquisition module's data receiving port to acquire real-world images, open the inertial measurement module's data receiving port to receive inertial measurement data, and then S20 to S50 can be executed to obtain the newly created berth map.

[0105] In one embodiment, S50, the current berth positioning information is obtained, the current real-scene image features, the patrol vehicle pose, and the berth positioning information are associated and bound, and a starting point is established using the patrol vehicle pose at the first frame as the initial pose to generate a berth map, including:

[0106] S510 detects and identifies the berth image, obtains the berth number, and obtains the berth location information based on the berth number;

[0107] Alternatively, obtain the patrol vehicle's location information, compare the patrol vehicle's location information with the location information list, and use the location information closest to the patrol vehicle's location information as the parking space location information;

[0108] Alternatively, berth location information can be obtained based on RFID tags.

[0109] In this embodiment, the acquisition of berth location information is not limited to the type of berth information; it can be GPS data, berth number, or RFID data, allowing the inspection vehicle's system to be compatible with multiple types of location information. Real-time acquisition of berth images, detection and recognition of the berth images, and parsing of the berth number are used to locate the berth and obtain its location information. Alternatively, the inspection vehicle's own GPS data can be acquired in real-time and compared with the berth's GPS information, selecting the nearest berth as the current berth to locate it and obtain its location information. Alternatively, berth marker data, such as the RFID tag of the berth, can be detected to locate the current berth and obtain its location information.

[0110] In one embodiment, after acquiring the real-scene image and inertial measurement information in S10, and before performing feature extraction on each frame of the real-scene image in S20 to obtain real-scene image features, and before performing feature matching between the real-scene image features at the current moment and the real-scene image features at the previous moment, the berth positioning method based on multi-sensor fusion further includes:

[0111] S101, determine whether the real-scene image at the current moment is the first frame image;

[0112] S102, if so, continue to acquire real-scene images to obtain multiple frames of real-scene images;

[0113] S103, if not, then perform feature matching between the real-scene image features at the current time and the real-scene image features at the previous time.

[0114] In this embodiment, after executing S10, it is determined whether the current real-scene image is the first frame. If it is the first frame, the process returns to S10 to continue acquiring the next frame of the real-scene image, and then performs corresponding feature parsing and extraction on the subsequent frames. If it is not the first frame, then at least two frames of the real-scene image have been acquired, and S30 can be executed to match the feature information of two adjacent frames and calculate the pose change of the image acquisition module relative to the previous moment.

[0115] In one embodiment, if the current pose of the image acquisition module cannot be calculated due to reasons such as errors in inertial measurement information or sensor unavailability, S10 can be executed to acquire the next frame of real-scene image, and the pose of the image acquisition module can be obtained in the next pose calculation process, thereby obtaining the pose of the inspection vehicle.

[0116] In one embodiment, S02, if a historical berth map exists, then berth location information is obtained based on the historical berth map, including:

[0117] S021, If ​​a historical berth map exists, obtain the current new real-scene image and extract features from the current new real-scene image to obtain the features of the current new real-scene image;

[0118] S022, Search the historical berth map for matching historical real-scene image features that match the current new real-scene image features;

[0119] S023, Based on the features of the matching historical real-world images, obtain the corresponding matching historical inspection vehicle pose, and use the matching historical inspection vehicle pose as the initial pose.

[0120] S024, starting from the initial pose, update the historical berth map according to the map update method to obtain the updated berth map, and obtain the berth positioning information based on the updated berth map.

[0121] In this embodiment, if a historical berth map exists, it is not necessary to rebuild the berth map; the historical berth map can be updated. The historical berth map includes real-scene image features, patrol vehicle pose, and berth positioning information. After obtaining the current new real-scene image, feature extraction is performed on the image to obtain the current new real-scene image features. Then, the feature information matching the current new real-scene image features is queried in the historical berth map. In one embodiment, according to the feature matching algorithm, the current new real-scene image features are compared and calculated with each real-scene image feature in the historical berth map. The matching historical patrol vehicle pose that is closest to the current patrol vehicle pose corresponding to the current new real-scene image features is obtained and used as the initial pose. Steps S20 to S50 in the above embodiment are executed as the new starting point to update the historical berth map and obtain the updated berth map.

[0122] The map update method includes two strategies. One is to update the historical berth map at preset fixed time intervals. The other is to update the historical berth map based on the degree of overlap between the features of the current new real-scene image and the features of the historical real-scene images stored in the historical berth map. When the feature overlap is lower than a set threshold, the surface map data has changed significantly and is no longer consistent with the actual real-scene image, requiring an update of the historical berth map.

[0123] The creation of the berth map is an iterative process. Real-scene image features, inspection vehicle positions, and berth positioning information can be continuously updated during the map iteration process, ensuring the accuracy of berth positioning achieved by the multi-sensor fusion-based berth positioning method provided by this invention.

[0124] In one embodiment, if no matching historical inspection vehicle position is found in the historical berth map, the image acquisition module continues to acquire the next frame of new real-scene image for feature analysis, and continues to execute steps S021 to S024 to try matching the initial position again.

[0125] In one embodiment, S02, if a historical berth map exists, then berth location information is obtained based on the historical berth map, including:

[0126] S021' If a historical berth map exists, the current new real-scene image is obtained, and each feature point in the current new real-scene image is extracted to form a current new feature point set, which includes multiple current new feature points;

[0127] S022', find the number of historical real-world images containing the current new feature point in the historical berth map, and use it as the first observation frame number;

[0128] S023': Find all historical real-view images adjacent to the current new real-view image in the historical berth map to form a set of adjacent historical real-view images;

[0129] S024', Determine whether there is a historical feature point in the adjacent historical real-scene image set that matches the current new feature point;

[0130] S025', if it exists, then find the number of adjacent historical real-scene images that match the current new feature point in the adjacent historical real-scene image set, and use it as the second observation frame number;

[0131] S026', compare the first observation frame number with the second observation frame number, retain the real scene image corresponding to the maximum observation frame number, update the historical berth map according to the feature points of the real scene image corresponding to the maximum observation frame number, and obtain the updated berth map.

[0132] If S027' does not exist, the new feature point will be added to the historical berth map for updating, and the updated berth map will be obtained.

[0133] In this embodiment, the current new real-scene image includes multiple feature points, forming a current new feature point set P. c ={p k |k∈[1,N]}. Find other points P in the historical berth map that contain the current new feature point, excluding the current new real-scene image. k The number of historical real-world images is used as the first observation frame number.

[0134] Find all historical real-scene images adjacent to the current new real-scene image in the historical berth map, forming a set S of adjacent historical real-scene images, and then find the feature point P in the set S that is adjacent to the current new real-scene image. k Matching feature points. If none exist, then the current new feature point P is selected. kThe data is added to the historical berth map to update it. If a historical berth exists, the number of adjacent historical real-scene images in the adjacent image set is used as the second observation frame number. The first and second observation frame numbers are compared, and the real-scene image with the largest number of observation frames is retained in the historical berth map. All real-scene images with smaller number of observation frames are deleted, thus updating the historical berth map. Therefore, based on the historical berth map, the map information is continuously updated using newly input real-scene images to obtain an updated berth map. Furthermore, berth location information is obtained based on the updated berth map.

[0135] In one embodiment, S02, if a historical berth map exists, the berth location information is obtained based on the historical berth map, further including:

[0136] S028': Before adding the current new feature points to the historical berth map, optimize the historical inspection vehicle pose and historical real-world image features of the historical berth map using the graph optimization algorithm to obtain the updated berth map.

[0137] In this embodiment, before adding the new feature points to the historical berth map, the accuracy of the berth positioning information in the historical berth map is further checked. Using a graph optimization algorithm, based on the historical berth map, the patrol vehicle pose at the time of real-scene image acquisition and the feature points of the historical real-scene images are optimized, and erroneous patrol vehicle poses and historical real-scene image feature points are deleted. This overall optimization of the berth map data results in an updated berth map.

[0138] In one embodiment, S70, after querying the parking space location information corresponding to the nearest inspection vehicle's position in the parking space map and using it as the parking space location information corresponding to the real-world image to be tested, the method further includes:

[0139] S810 sends a stop data acquisition command to the image acquisition module and the inertial measurement module;

[0140] S820 stores the berth map.

[0141] In this embodiment, after receiving the stop signal, the inspection vehicle's system shuts down the image acquisition module and the inertial measurement module, stops data acquisition, and encodes and stores the current berth map data in the local storage space.

[0142] In one embodiment, after storing the berth map in S820, the main control system of the inspection vehicle exits.

[0143] Please see Figure 2This invention provides a berth positioning system 100 based on multi-sensor fusion. The berth positioning system 100 includes a data receiving module 10, a feature matching module 20, an image pose calculation module 30, an inertial measurement pose calculation module 40, a berth map generation module 50, a pose query module 60, and a berth positioning module 70. The data receiving module 10 is used to acquire real-scene images and inertial measurement information. The feature matching module 20 is used to extract features from each frame of the real-scene image, acquire real-scene image features, and perform feature matching between the real-scene image features at the current moment and the real-scene image features at the previous moment. The image pose calculation module 30 is used to calculate the pose of the image acquisition module at the current moment based on the real-scene image features at the current moment and the real-scene image features at the previous moment if the matching is successful.

[0144] The inertial measurement module 40 is used to obtain the relative positional relationship between the inertial measurement module and the image acquisition module if the matching fails. Based on the relative positional relationship and the inertial measurement information from the previous moment, it calculates the image acquisition module pose at the current moment, which is then used as the patrol vehicle pose. The berth map generation module 50 is used to obtain the berth positioning information at the current moment, associate and bind the real-scene image features, patrol vehicle pose, and berth positioning information at the current moment, and establish a starting point using the patrol vehicle pose from the first frame as the initial pose to generate a berth map.

[0145] The pose query module 60 is used to extract features from the real-world image under test, obtain the features of the real-world image under test, and obtain the pose of the inspection vehicle under test based on the features of the real-world image under test. It then compares the pose of the inspection vehicle under test with the poses of all inspection vehicles in the berth map to obtain the pose of the nearest inspection vehicle. The berth positioning module 70 is used to query the berth positioning information corresponding to the pose of the nearest inspection vehicle in the berth map, and uses this as the berth positioning information corresponding to the real-world image under test. The image acquisition module is used to acquire real-world images, and the inertial measurement module is used to measure inertial measurement information.

[0146] In this embodiment, the description of the data receiving module 10 can be found in the description of S10 in the above embodiment. The description of the feature matching module 20 can be found in the description of S20 in the above embodiment. The description of the image pose calculation module 30 can be found in the description of S30 in the above embodiment. The description of the inertial measurement pose calculation module 40 can be found in the description of S40 in the above embodiment. The description of the berth map generation module 50 can be found in the description of S50 in the above embodiment. The description of the pose query module 60 can be found in the description of S60 in the above embodiment. The description of the berth positioning module 70 can be found in the description of S70 in the above embodiment.

[0147] In one embodiment, the berth positioning system 100 based on multi-sensor fusion further includes a map existence determination module, a map existence determination result module one, and a map existence determination result module two. The map existence determination module is used to determine whether a historical berth map exists. The map existence determination result module one is used to obtain berth positioning information based on the historical berth map if one exists. The map existence determination result module two is used to execute the image acquisition module to acquire real-scene images if no historical berth map exists.

[0148] In this embodiment, the description of the map existence determination module can be found in the description of S01 in the above embodiment. The description of the first map existence determination result module can be found in the description of S02 in the above embodiment. The description of the second map existence determination result module can be found in the description of S03 in the above embodiment.

[0149] In one embodiment, the berth map generation module 50 includes a berth location information acquisition module one, a berth location information acquisition module two, or a berth location information acquisition module three. The berth location information acquisition module one is used to detect and identify berth images, obtain berth numbers, and obtain berth location information based on the berth numbers. The berth location information acquisition module two is used to obtain patrol vehicle location information, compare the patrol vehicle location information with a location information list, and use the location information closest to the patrol vehicle location information as the berth location information. The berth location information acquisition module three is used to obtain berth location information based on radio frequency identification tags.

[0150] In this embodiment, the relevant descriptions of the berth positioning information acquisition module one, the berth positioning information acquisition module two, or the berth positioning information acquisition module three can be referred to the relevant descriptions of S510 in the above embodiments.

[0151] In one embodiment, the berth positioning system 100 based on multi-sensor fusion further includes a first frame image judgment module, an image judgment result module one, and an image judgment result module two. The first frame image judgment module is used to determine whether the current real-scene image is the first frame image. The image judgment result module one is used to, if yes, continue acquiring real-scene images to obtain multiple frames of real-scene images. The image judgment result module two is used to, if no, perform feature extraction on each frame of real-scene image to obtain real-scene image features, and perform feature matching between the current real-scene image features and the previous real-scene image features.

[0152] In this embodiment, the description of the first frame image judgment module can be referred to the description of S101 in the above embodiment. The description of the first image judgment result module can be referred to the description of S102 in the above embodiment. The description of the second image judgment result module can be referred to the description of S103 in the above embodiment.

[0153] In one embodiment, the map existence determination module includes a current new real-scene image feature acquisition module, a historical real-scene image feature acquisition module, an initial pose acquisition module, and a berth map update module. The current new real-scene image feature acquisition module acquires the current new real-scene image if a historical berth map exists, and extracts features from the current new real-scene image to obtain its features. The historical real-scene image feature acquisition module queries the historical berth map for matching historical real-scene image features that match the current new real-scene image features. The initial pose acquisition module obtains the corresponding historical patrol vehicle pose based on the historical real-scene image features and uses the historical patrol vehicle pose as the initial pose. The berth map update module updates the historical berth map using the initial pose as the starting point, according to a map update method, to obtain an updated berth map, and obtains berth positioning information based on the updated berth map.

[0154] In this embodiment, the description of the current new real-scene image feature acquisition module can be found in the description of S021 in the above embodiment. The description of the historical real-scene image feature acquisition module can be found in the description of S022 in the above embodiment. The description of the initial pose acquisition module can be found in the description of S023 in the above embodiment. The description of the berth map update module can be found in the description of S024 in the above embodiment.

[0155] In one embodiment, the map existence determination module one includes a current new feature point set acquisition module, a first observation frame count acquisition module, an adjacent historical real-scene image set acquisition module, an adjacent historical set feature matching determination module, a second observation frame count acquisition module, a berth map update module two, and a berth map update module three. The current new feature point set acquisition module is used to acquire the current new real-scene image if a historical berth map exists, and extract each feature point from the current new real-scene image to form a current new feature point set, which includes multiple current new feature points. The first observation frame count acquisition module is used to find the number of historical real-scene images containing the current new feature points in the historical berth map, which is used as the first observation frame count.

[0156] The adjacent historical real-scene image set acquisition module is used to find all historical real-scene images adjacent to the current new real-scene image in the historical berth map, forming an adjacent historical real-scene image set. The adjacent historical set feature matching judgment module is used to determine whether there is a historical feature point in the adjacent historical real-scene image set that matches the current new feature point. The second observation frame number acquisition module is used to find the number of adjacent historical real-scene images in the adjacent historical real-scene image set that match the current new feature point if they exist, and use this number as the second observation frame number. The berth map update module two is used to compare the first observation frame number with the second observation frame number, retain the real-scene image corresponding to the maximum observation frame number, and update the historical berth map according to the feature points of the real-scene image corresponding to the maximum observation frame number to obtain the updated berth map. The berth map update module three is used to add the current new feature point to the historical berth map for updating if it does not exist, to obtain the updated berth map.

[0157] In this embodiment, the description of the current new feature point set acquisition module can refer to the description of S021' in the above embodiment. The description of the first observation frame acquisition module can refer to the description of S022' in the above embodiment. The description of the adjacent historical real-scene image set acquisition module can refer to the description of S023' in the above embodiment. The description of the adjacent historical set feature matching judgment module can refer to the description of S024' in the above embodiment. The description of the second observation frame acquisition module can refer to the description of S025' in the above embodiment. The description of the second berth map update module can refer to the description of S026' in the above embodiment. The description of the third berth map update module can refer to the description of S027' in the above embodiment.

[0158] In one embodiment, the berth positioning system 100 based on multi-sensor fusion further includes a command sending module and a storage module. The command sending module is used to send a stop data acquisition command to the image acquisition module and the inertial measurement module. The storage module is used to store the berth map.

[0159] In this embodiment, the description of the instruction sending module can be found in the description of S810 in the above embodiment. The description of the storage module can be found in the description of S820 in the above embodiment.

[0160] In one embodiment, a berth positioning system 100 based on multi-sensor fusion is installed on an inspection vehicle to receive real-scene images sent by an image acquisition module and inertial measurement information sent by an inertial measurement module, for use in berth positioning.

[0161] In the various embodiments described above, the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process may be rearranged without departing from the scope of this disclosure. The appended method claims provide elements of various steps in an exemplary order and are not intended to limit the scope to a specific order or hierarchy.

[0162] Those skilled in the art will also understand that the various illustrative logical blocks, modules, and steps listed in the embodiments of the present invention can be implemented by electronic hardware, computer software, or a combination of both. To clearly demonstrate the interchangeability of hardware and software, the functions of the various illustrative components, modules, and steps described above have been generally described. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functions using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of the present invention.

[0163] The various illustrative logic blocks or modules described in the embodiments of this invention can be implemented or operate the described functions using a general-purpose processor, digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor can be a microprocessor; alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented using a combination of computing devices, such as a digital signal processor and a microprocessor, multiple microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0164] The steps of the methods or algorithms described in the embodiments of this invention can be directly embedded in hardware, a software module executed by a processor, or a combination of both. The software module can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and storage medium can be housed in an ASIC, which can be housed in a user terminal. Optionally, the processor and storage medium can also be housed in different components of the user terminal.

[0165] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A berth positioning method based on multi-sensor fusion, characterized in that, include: Acquire real-world images and inertial measurement information; Feature extraction is performed on each frame of the real-scene image to obtain real-scene image features, and feature matching is performed between the real-scene image features at the current moment and the real-scene image features at the previous moment. If the match is successful, the pose of the image acquisition module at the current moment is calculated based on the real-scene image features at the current moment and the real-scene image features at the previous moment. If the matching fails, the relative positional relationship between the inertial measurement module and the image acquisition module is obtained, and the image acquisition module pose at the current moment is calculated based on the relative positional relationship and the inertial measurement information at the previous moment. The image acquisition module pose at the current moment is the inspection vehicle pose at the current moment. Obtain the berth positioning information at the current moment, associate and bind the real scene image features, the patrol vehicle pose, and the berth positioning information at the current moment, and establish the starting point with the patrol vehicle pose at the first frame as the initial pose to generate a berth map. Acquire a real-world image to be tested, extract features from the real-world image to obtain the features of the real-world image to be tested, and obtain the position pose of the inspection vehicle to be tested based on the features of the real-world image to be tested, and compare the position pose of the inspection vehicle to be tested with all the position poses of the inspection vehicles in the parking space map to obtain the position pose of the inspection vehicle closest to it. The location information of the nearest inspection vehicle position is queried in the berth map and used as the location information of the berth corresponding to the real scene image to be tested. The image acquisition module is used to acquire the real-scene image, and the inertial measurement module is used to measure the inertial measurement information. Before the image acquisition module acquires the real-scene image, the method further includes: Determine if a historical berth map exists; If the historical berth map exists, then obtain the berth location information based on the historical berth map; If the historical berth map does not exist, the image acquisition module will be executed to acquire real-scene images; If the historical berth map exists, then obtaining berth location information based on the historical berth map includes: If the historical berth map exists, the current new real-scene image is obtained, and each feature point in the current new real-scene image is extracted to form a current new feature point set, which includes multiple current new feature points; The number of historical real-world images containing the current new feature point is found in the historical berth map and used as the first observation frame number; Find all historical real-view images adjacent to the current new real-view image in the historical berth map to form an adjacent historical real-view image set; Determine whether there exists a historical feature point in the adjacent historical real-scene image set that matches the current new feature point; If it exists, then find the number of adjacent historical real-scene images that match the current new feature point in the adjacent historical real-scene image set, and use it as the second observation frame number; The first observation frame number is compared with the second observation frame number. The real-scene image corresponding to the maximum observation frame number is retained. The historical berth map is updated according to the feature points of the real-scene image corresponding to the maximum observation frame number to obtain the updated berth map. If it does not exist, the current new feature point is added to the historical berth map for updating, and the updated berth map is obtained.

2. The berth positioning method based on multi-sensor fusion according to claim 1, characterized in that, The step of obtaining the parking space positioning information at the current moment, associating and binding the real-scene image features, the inspection vehicle pose, and the parking space positioning information at the current moment, and establishing a starting point with the inspection vehicle pose at the first frame as the initial pose to generate a parking space map includes: The berth image is detected and identified to obtain the berth number, and the berth location information is obtained based on the berth number; Alternatively, obtain the location information of the inspection vehicle, compare the location information of the inspection vehicle with the location information list, and take the location information that is closest to the location information of the inspection vehicle as the parking space location information; Alternatively, the berth location information can be obtained based on the radio frequency identification tag.

3. The berth positioning method based on multi-sensor fusion according to claim 1, characterized in that, After the image acquisition module acquires real-scene images, before performing feature extraction on each frame of the real-scene image to obtain real-scene image features, and before performing feature matching between the real-scene image features at the current moment and the real-scene image features at the previous moment, the method further includes: Determine whether the real-scene image at the current moment is the first frame image; If so, continue to acquire the real-scene images to obtain multiple frames of the real-scene images; If not, then the feature extraction of each frame of the real-scene image is performed to obtain the real-scene image features, and the real-scene image features at the current moment are matched with the real-scene image features at the previous moment.

4. The berth positioning method based on multi-sensor fusion according to claim 1, characterized in that, If the historical berth map exists, then obtaining berth location information based on the historical berth map includes: If the historical berth map exists, then the current new real-scene image is obtained, and features are extracted from the current new real-scene image to obtain the features of the current new real-scene image; Search the historical berth map for matching historical real-scene image features that match the current new real-scene image features; Based on the features of the matched historical real-world images, the corresponding matched historical inspection vehicle pose is obtained, and the matched historical inspection vehicle pose is used as the initial pose. Using the initial pose as the starting point, the historical berth map is updated according to the map update method to obtain an updated berth map, and the berth positioning information is obtained based on the updated berth map.

5. The berth positioning method based on multi-sensor fusion according to claim 1, characterized in that, After obtaining the berth positioning information by querying the nearest pose information in the berth map, the method further includes: Send a stop data acquisition command to the image acquisition module and the inertial measurement module; Store the berth map.

6. A berth positioning system based on multi-sensor fusion, characterized in that, include: The data receiving module is used to acquire real-scene images and inertial measurement information; The feature matching module is used to extract features from each frame of the real-scene image, obtain real-scene image features, and perform feature matching between the real-scene image features at the current moment and the real-scene image features at the previous moment. The image pose calculation module is used to calculate the pose of the image acquisition module at the current moment based on the real scene image features at the current moment and the real scene image features at the previous moment if the match is successful. An inertial measurement module is used to obtain the relative positional relationship between the inertial measurement module and the image acquisition module if the matching fails, and to calculate the image acquisition module pose at the current moment based on the relative positional relationship and the inertial measurement information at the previous moment, wherein the image acquisition module pose at the current moment is the inspection vehicle pose at the current moment. The berth map generation module is used to obtain the berth positioning information at the current moment, associate and bind the real scene image features, the patrol vehicle position pose, and the berth positioning information at the current moment, and establish the starting point with the patrol vehicle position pose at the first frame moment to generate the berth map. The pose query module is used to extract features from the real-scene image to be tested, obtain the features of the real-scene image to be tested, obtain the pose of the inspection vehicle to be tested based on the features of the real-scene image to be tested, and compare the pose of the inspection vehicle to be tested with all the poses of the inspection vehicles in the parking space map to obtain the pose of the nearest inspection vehicle. The berth positioning module is used to query the berth positioning information corresponding to the nearest inspection vehicle position in the berth map, and use it as the berth positioning information corresponding to the real scene image to be tested. The image acquisition module is used to acquire the real-scene image, and the inertial measurement module is used to measure the inertial measurement information. The system also includes: The map has a judgment module used to determine whether a historical berth map exists; The first module for determining the existence of a map is used to obtain berth location information based on the historical berth map if the historical berth map exists. The second map existence judgment module is used to execute the image acquisition module to acquire real-scene images if the historical berth map does not exist. The map existence determination module one includes: The current new feature point set acquisition module is used to acquire the current new real-scene image if the historical berth map exists, and extract each feature point in the current new real-scene image to form a current new feature point set, which includes multiple current new feature points; The first observation frame acquisition module is used to find the number of historical real-world images containing the current new feature point in the historical berth map, and use it as the first observation frame number. The adjacent historical real-scene image set acquisition module is used to find all historical real-scene images adjacent to the current new real-scene image in the historical berth map, and form an adjacent historical real-scene image set; The adjacent historical set feature matching judgment module is used to determine whether there is a historical feature point in the adjacent historical real scene image set that matches the current new feature point; The second observation frame acquisition module is used to find, if it exists, the number of adjacent historical real-scene images that match the current new feature point in the adjacent historical real-scene image set, and use it as the second observation frame number. The second module for updating the berth map is used to compare the first observation frame number with the second observation frame number, retain the real-scene image corresponding to the maximum observation frame number, and update the historical berth map according to the feature points of the real-scene image corresponding to the maximum observation frame number to obtain the updated berth map. The third module for updating the berth map is used to add the current new feature point to the historical berth map for updating if it does not exist, thereby obtaining the updated berth map.

7. The berth positioning system based on multi-sensor fusion according to claim 6, characterized in that, The berth map generation module includes: The berth location information acquisition module 1 is used to detect and identify berth images, obtain berth numbers, and obtain berth location information based on the berth numbers; Alternatively, the second berth positioning information acquisition module is used to acquire the patrol vehicle positioning information, compare the patrol vehicle positioning information with the positioning information list, and take the positioning information closest to the patrol vehicle positioning information as the berth positioning information; Alternatively, the berth location information acquisition module three is used to acquire the berth location information based on the radio frequency identification tag.

8. The berth positioning system based on multi-sensor fusion according to claim 6, characterized in that, The system also includes: The first frame image determination module is used to determine whether the real-scene image at the current moment is the first frame image; Image judgment result module one is used to continue acquiring the real scene image if the result is positive, thereby obtaining multiple frames of the real scene image; The second image judgment result module is used to perform feature extraction on each frame of the real scene image if the condition is not met, obtain the real scene image features, and perform feature matching between the real scene image features at the current moment and the real scene image features at the previous moment.

9. The berth positioning system based on multi-sensor fusion according to claim 6, characterized in that, The map existence determination module one includes: The current new real-scene image feature acquisition module is used to acquire the current new real-scene image if the historical berth map exists, and to extract features from the current new real-scene image to obtain the current new real-scene image features. The historical real-scene image feature acquisition module is used to query the historical berth map for matching historical real-scene image features that match the current new real-scene image features; The initial pose acquisition module is used to obtain the corresponding historical patrol vehicle pose based on the features of the historical matching images, and to use the historical matching patrol vehicle pose as the initial pose. The berth map update module 1 is used to update the historical berth map according to the map update method, starting from the initial pose, to obtain an updated berth map, and to obtain the berth positioning information based on the updated berth map.

10. The berth positioning system based on multi-sensor fusion according to claim 6, characterized in that, The system also includes: The instruction sending module is used to send a stop data acquisition instruction to the image acquisition module and the inertial measurement module; A storage module is used to store the berth map.

Images