Image matching method and device for assisting visual positioning

Abstract

The invention provides an image matching method and device for assisting visual positioning, and the method comprises the steps: calculating the image entropy of a current starting frame when a starting point positioning signal is received, comparing the image entropy with a first threshold value, and enabling the image of the current starting frame to enter cosine similarity matching after a threshold value condition is satisfied. And then, screening out a scene similar to the current starting frame by calculating a sensing result of the current starting frame and cosine similarity of each image frame in the mapping scene, and finally determining the position and attitude of the starting point in the mapping by performing point cloud ICP registration on the current starting frame and each similar frame in the similar scene. According to the method, any point can be positioned in a scene meeting conditions in the established map, the initial point can be positioned more quickly and flexibly, the flexibility and the adaptability are high, and the application range and the robustness of the system are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent connected vehicle technology, and in particular to an image matching method and apparatus for assisting visual positioning. Background Technology

[0002] SLAM (Simultaneous Localization and Mapping) technology is widely used in fields such as autonomous driving, robot navigation, and augmented reality. In these applications, SLAM collects environmental information through sensors (such as cameras and LiDAR), builds maps in real time, and performs localization, enabling devices to move autonomously in unknown environments.

[0003] In visual SLAM technology, vehicle localization typically requires the starting position to be close to the starting point during mapping, with minimal attitude and position errors. This method relies on matching the current localization starting point with the mapping starting point to ensure accurate subsequent localization. However, this approach has limitations: the vehicle must begin localization near the mapping starting point, almost perfectly replicating its attitude and position at that time. If it deviates from the mapping starting point, localization accuracy may decrease, and it cannot support localization from any point on the map. Therefore, existing solutions lack sufficient flexibility and adaptability, limiting the system's application scope and robustness. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an image matching method and apparatus for assisting visual positioning, so as to solve the problem of difficult and low-accuracy positioning of the map construction starting point.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of this invention discloses an image matching method for assisted visual localization, the method comprising:

[0007] When the starting point positioning signal is received, the image entropy of the current starting frame is calculated;

[0008] If the image entropy is greater than the first threshold, the semantic segmentation image of the current starting frame surround stitching map is converted into the bag-of-words model vector of the current starting frame, and the semantic segmentation images of each image frame in the mapping data are converted into the bag-of-words model vector of each image frame.

[0009] Based on the bag-of-words model vectors of each image frame and the bag-of-words model vector of the current starting frame, select similar frames to the current starting frame from each image frame;

[0010] The semantic segmentation image of the current starting frame surround stitching image is converted into a point cloud to obtain the target point cloud, and the semantic segmentation images of multiple similar frames are converted into point clouds to obtain multiple registered point clouds.

[0011] Each registered point cloud is registered with the target point cloud to obtain a pose matrix, and the overlap between the current starting frame and multiple similar frames is calculated.

[0012] The similar frame corresponding to the highest degree of overlap is determined as the final starting point, and the pose matrix of the similar frame is used as the pose of the current starting frame in the mapping.

[0013] Preferably, the step of converting the semantic segmentation image of the current starting frame lookaround stitched image into the bag-of-words model vector of the current starting frame includes:

[0014] Dilate each non-zero semantic element in the semantic segmentation image of the current starting frame surround stitching image;

[0015] The semantic segmentation image after dilation is downsampled using an image resizing function;

[0016] The downsampled semantic segmentation image is subjected to an erosion operation to obtain the semantic segmentation image to be transformed;

[0017] The pixel value of each non-zero pixel in the semantic segmentation image to be transformed is converted into a string type and stored in a hash table to obtain the bag-of-words model vector of the current starting frame.

[0018] Preferably, the step of selecting similar frames from the image frames based on the bag-of-words model vectors of each image frame and the bag-of-words model vector of the current starting frame includes:

[0019] Based on the bag-of-words model vectors of each image frame and the bag-of-words model vector of the current starting frame, calculate multiple cosine similarities;

[0020] Image frames corresponding to multiple target cosine similarities are marked as similar frames, where the target cosine similarity is the cosine similarity among the multiple cosine similarities that is greater than a second threshold.

[0021] Preferably, the calculation of multiple cosine similarities based on the bag-of-words model vectors of each image frame and the bag-of-words model vector of the current starting frame includes:

[0022] For each image frame in each image frame, determine whether the bag-of-words model vector of the image frame is equal in size to the bag-of-words model vector of the current starting frame;

[0023] If they are not equal in size, the number of lane line semantics in the larger bag-of-words model vector is deleted, and the process returns to the step of determining whether the bag-of-words model vector of the image frame is equal in size to the bag-of-words model vector of the current starting frame.

[0024] If the sizes are equal, the cosine similarity between the bag-of-words model vector of the image frame and the bag-of-words model vector of the current starting frame is calculated to obtain multiple cosine similarities.

[0025] Preferably, calculating the overlap between the current starting frame and multiple similar frames includes:

[0026] The corresponding similar frames of the pose matrix are rotated and translated.

[0027] Calculate the overlap between similar frames after rotation and translation processing and the current starting frame.

[0028] A second aspect of the present invention discloses an image matching device for assisting visual positioning, the device comprising:

[0029] The calculation unit is used to calculate the image entropy of the current starting frame when the starting point positioning signal is received;

[0030] The conversion unit is used to convert the semantic segmentation image of the current starting frame surround stitching map into the bag-of-words model vector of the current starting frame if the image entropy is greater than a first threshold, and to convert the semantic segmentation image of each image frame in the mapping data into the bag-of-words model vector of each image frame.

[0031] The selection unit is used to select similar frames to the current starting frame from each image frame based on the bag-of-words model vector of each image frame and the bag-of-words model vector of the current starting frame.

[0032] The image-to-point-cloud unit is used to perform image-to-point-cloud operations on the semantic segmentation image of the current starting frame surround stitching image to obtain the target point cloud, and to perform image-to-point-cloud operations on the semantic segmentation images of multiple similar frames to obtain multiple registered point clouds.

[0033] The point cloud registration unit is used to register each of the registered point clouds with the target point cloud to obtain a pose matrix and calculate the overlap between the current starting frame and multiple similar frames.

[0034] The determining unit is used to determine the similar frame corresponding to the highest overlap among all overlap degrees as the final starting point, and to use the pose matrix of the similar frame as the pose of the current starting frame in the mapping.

[0035] Preferably, the conversion unit includes:

[0036] The dilation module is used to dilate each non-zero semantic element in the semantic segmentation image of the current starting frame lookaround stitching image.

[0037] The downsampling module is used to downsample the semantic segmentation image after dilation using an image resizing function;

[0038] The erosion module is used to perform an erosion operation on the downsampled semantic segmentation image to obtain the semantic segmentation image to be converted.

[0039] The storage module is used to convert the pixel value of each non-zero pixel in the semantic segmentation image to be transformed into a string type and store it in a hash table to obtain the bag-of-words model vector of the current starting frame.

[0040] Preferably, the selection unit includes:

[0041] The calculation module is used to calculate multiple cosine similarities based on the bag-of-words model vectors of each image frame and the bag-of-words model vector of the current starting frame;

[0042] The labeling module is used to label image frames corresponding to multiple target cosine similarities as similar frames, wherein the target cosine similarity is the cosine similarity among the multiple cosine similarities that is greater than a second threshold.

[0043] Preferably, the computing module includes:

[0044] The judgment submodule is used to determine, for each image frame in each image frame, whether the bag-of-words model vector of the image frame is equal in size to the bag-of-words model vector of the current starting frame;

[0045] The deletion submodule is used to delete the number of lane line semantics in the larger bag-of-words model vector if they are not equal in size, and then return to execute the judgment submodule.

[0046] The calculation submodule is used to calculate the cosine similarity between the bag-of-words model vector of the image frame and the bag-of-words model vector of the current starting frame if the sizes are equal, and obtain multiple cosine similarities.

[0047] Preferably, the point cloud registration unit is specifically used for:

[0048] Perform rotation and translation processing on the corresponding similar frames of the pose matrix; calculate the overlap between the similar frames after rotation and translation processing and the current starting frame.

[0049] Based on the image matching method and apparatus for assisted visual positioning provided by the above embodiments of the present invention, when a starting point positioning signal is received, the image entropy of the current starting frame is calculated and compared with a first threshold. If the threshold condition is met, the current starting frame image enters cosine similarity matching. Next, by calculating the cosine similarity between the perception result of the current starting frame and each image frame in the mapped scene, scenes similar to the current starting frame are selected. Then, by performing point cloud ICP registration between the current starting frame and each similar frame in the similar scenes, the position and pose of the starting point in the mapped scene are finally determined. The present invention enables arbitrary point positioning in scenes that meet the conditions in the already mapped scene, and can complete starting point positioning more quickly and flexibly, exhibiting strong flexibility and adaptability, significantly increasing the application range and robustness of the system. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 A flowchart of an image matching method for assisted visual positioning provided in an embodiment of the present invention;

[0052] Figure 2 The first semantic segmentation image provided in the embodiments of the present invention;

[0053] Figure 3 The second semantic segmentation image provided in this embodiment of the invention;

[0054] Figure 4 This is a semantic segmentation image of the current starting frame surround view stitching diagram provided in an embodiment of the present invention;

[0055] Figure 5 This is a semantic segmentation image after dilation operation provided in an embodiment of the present invention;

[0056] Figure 6 This is a downsampled semantic segmentation image provided in an embodiment of the present invention;

[0057] Figure 7 This is a semantic segmentation image to be converted obtained after performing an erosion operation, as provided in an embodiment of the present invention.

[0058] Figure 8 This is a semantic segmentation image of a vehicle during driving, provided in an embodiment of the present invention.

[0059] Figure 9This is another semantic segmentation image during the driving process provided in an embodiment of the present invention;

[0060] Figure 10 This is another semantic segmentation image during the driving process provided in an embodiment of the present invention;

[0061] Figure 11 This is a structural block diagram of an image matching device for assisted visual positioning provided in an embodiment of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0064] As the background technology shows, current vehicle localization must begin near the starting point during mapping, and the vehicle's attitude and position at that time must be replicated. Deviation from the starting point leads to decreased localization accuracy and cannot support localization from any point on the map. Therefore, existing solutions lack flexibility and adaptability, limiting the system's application scope and robustness.

[0065] Therefore, embodiments of the present invention provide an image matching method and apparatus for assisted visual localization. When a starting point localization signal is received, the image entropy of the current starting frame is calculated and compared with a first threshold. If the threshold condition is met, the current starting frame image enters cosine similarity matching. Next, by calculating the cosine similarity between the perception result of the current starting frame and each image frame in the mapped scene, scenes similar to the current starting frame are selected. Then, by performing point cloud ICP registration between the current starting frame and each similar frame in the similar scenes, the position and pose of the starting point in the mapped scene are finally determined. The present invention enables arbitrary point localization in scenes that meet the conditions in the already mapped scene, and can complete the starting point localization more quickly and flexibly, exhibiting strong flexibility and adaptability, significantly increasing the application range and robustness of the system.

[0066] See Figure 1 The diagram illustrates a flowchart of an image matching method for assisted visual localization provided by an embodiment of the present invention. The method includes:

[0067] Step S101: When the starting point positioning signal is received, calculate the image entropy of the current starting frame.

[0068] It should be noted that the car is equipped with a fisheye camera (surround view camera).

[0069] Understandably, when locating a starting point in an underground parking lot without GPS support, the location within the map can only be determined by the panoramic stitched image of the starting frame. Therefore, the semantic richness of the starting frame image is crucial for location determination. If the semantics of the starting frame image are insufficient, it is likely to lead to location errors. To improve the accuracy of starting point positioning, the concept of image entropy is introduced to determine whether the current starting frame is suitable for initializing the positioning.

[0070] In the specific implementation of step S101, when a starting point positioning signal indicating the start of starting point positioning is received, the image entropy of the current starting frame is calculated.

[0071] It should be noted that image entropy is a statistical characteristic used to represent the average number of bits in a set of gray levels in an image, reflecting the amount of information in the image. Its calculation is shown in formula (1):

[0072] (1)

[0073] Where H represents image entropy; p i This represents the probability of grayscale value i appearing in the image.

[0074] It is understood that, as illustrated in the embodiments of the present invention Figure 2 and Figure 3 For example. Figure 2 The image entropy is 0.44833025; Figure 3 The image entropy is 0.71360278. From Figure 2 and Figure 3 It can be seen from this that Figure 3 Its semantic richness is higher than Figure 2 .

[0075] It should be noted that after calculating the image entropy of the current starting frame, it is determined whether the image entropy is greater than a first threshold. If it is, step S102 is executed; if the image entropy is not greater than the first threshold, a prompt message is output. For example, a prompt message such as "The semantic richness of the current positioning environment is low, and the vehicle needs to continue traveling a certain distance" is output.

[0076] The first threshold can be set in advance based on existing empirical values.

[0077] Step S102: If the image entropy is greater than the first threshold, the semantic segmentation image of the current starting frame surround stitching map is converted into the bag-of-words model vector of the current starting frame, and the semantic segmentation images of each image frame in the mapping data are converted into the bag-of-words model vector of each image frame.

[0078] In the specific implementation step S102, when the image entropy is greater than the first threshold, the semantic segmentation image of the current starting frame surround stitching map is converted into the bag-of-words model vector of the current starting frame, and the semantic segmentation images of each image frame in the mapping data are converted into the bag-of-words model vector of each image frame.

[0079] It should be noted that the semantic segmentation image of the current starting frame surround stitching map and the semantic segmentation images of each image frame in the mapping data are specifically obtained through inference by a deep learning model.

[0080] Understandably, to improve computational efficiency, the size of the image look-around stitched image is 512×512. However, calculating the bag-of-words model vector requires traversing the image, which significantly impacts the overall computational speed of the algorithm. Therefore, downsampling of the image before vector calculation is necessary. However, downsampling may lead to the loss of some semantic information in the image, negatively affecting the algorithm's performance. To address this issue, this embodiment of the invention proposes a transformation process (processes A1 to A4) that ensures that image downsampling does not result in the loss of crucial semantic information, thereby optimizing the algorithm's performance.

[0081] The process of converting the semantic segmentation image of the current starting frame lookaround stitched image into the bag-of-words model vector of the current starting frame is as follows (processes A1 to A4):

[0082] Process A1: Dilate each non-zero semantic segment in the semantic segmentation image of the current starting frame surround stitching image.

[0083] In the specific implementation process A1, the `dilate` function from the OpenCV library is used to perform dilation operations on each non-zero semantic element in the semantic segmentation image of the current starting frame's lookaround stitched image. The dilation operation is performed by convolving the image with a dilation kernel. Specifically, first, the maximum pixel value within the kernel's coverage area is calculated and assigned to the pixel value at the kernel's center point. Then, the entire image is progressively traversed using the kernel to complete the image dilation process.

[0084] It is understandable that the semantic segmentation image of the current starting frame surround stitching is, for example... Figure 4 As shown; the semantic segmentation image after dilation operation, for example Figure 5 As shown, where, Figure 5 The corresponding expansion core is 30×30.

[0085] It should be noted that the purpose of dilating the image is to expand the non-zero semantic regions within the image, thereby ensuring that certain semantic regions are not ignored or lost during subsequent downsampling due to their small number. In this way, the dilated image can effectively preserve important semantic information, providing a guarantee for subsequent downsampling.

[0086] Process A2: Downsample the semantic segmentation image after dilation using an image resizing function.

[0087] In the specific implementation process A2, the resize function (i.e., the image resizing function) in the OpenCV library is used to downsample the semantic segmentation image after the dilation operation.

[0088] It is important to note that, typically, to ensure the quality of the downsampled image, the default resize function uses bilinear interpolation. However, bilinear interpolation can introduce incorrect semantic information. Therefore, in this embodiment of the invention, nearest-neighbor interpolation is preferred to avoid this problem.

[0089] It is understandable that the downsampled semantic segmentation image, for example... Figure 6 As shown, in Figure 6 In this process, the image is downsampled to a size of 30×30.

[0090] Process A3: Perform an erosion operation on the downsampled semantic segmentation image to obtain the semantic segmentation image to be converted.

[0091] Understandably, the dilation operation expands the number of non-zero semantics in the image, while the erosion operation aims to uniformly reduce the number of non-zero semantics after downsampling, thereby reducing the number of semantics in the bag-of-words vector and improving computational speed. The principle of erosion is similar to dilation; it uses the minimum value within the area covered by the erosion kernel as the pixel value of the kernel's center point. Then, the erosion kernel traverses the entire image to complete the erosion operation.

[0092] It should be noted that the semantic segmentation image to be transformed after the erosion operation is, for example... Figure 7 As shown, in Figure 7 In this case, the corrosion nuclei are set to 2×2.

[0093] Process A4: Convert the pixel value of each non-zero pixel in the semantic segmentation image to be transformed into a string type and store it in a hash table to obtain the bag-of-words model vector of the current starting frame.

[0094] For example, the bag-of-words model vector is: 11111133333333333333333333333331111111111111111111111111111111111222222222222111111111555555555555111111111111.

[0095] In the specific implementation process A4, the pixel value of each non-zero pixel in the semantic segmentation image to be transformed is converted into a string type and stored in a hash table to obtain a hash table of pixel type and number in the semantic segmentation image, which is the bag-of-words model vector of the current starting frame.

[0096] It is understandable that the specific implementation principle of converting the semantic segmentation image of each image frame in the mapping data into the bag-of-words model vector of each image frame is consistent with the implementation principle of processes A1 to A4. That is, for the semantic segmentation image of each image frame in the mapping data, it is converted in the way shown in processes A1 to A4 to obtain the bag-of-words model vector of each image frame.

[0097] Step S103: Based on the bag-of-words model vectors of each image frame and the bag-of-words model vector of the current starting frame, select similar frames to the current starting frame from each image frame.

[0098] In the specific implementation step S103, multiple cosine similarities are calculated based on the bag-of-words model vectors of each image frame and the bag-of-words model vector of the current starting frame; the image frames corresponding to the multiple target cosine similarities are marked as similar frames.

[0099] The target cosine similarity is the cosine similarity among multiple cosine similarities that is greater than the second threshold. The second threshold can be adjusted based on a large number of test results in actual parking lot scenarios to ensure that high accuracy is maintained while ensuring efficiency.

[0100] In some specific embodiments, multiple cosine similarities are sorted, and cosine similarities greater than a second threshold are marked as target cosine similarities. The image frames corresponding to the target cosine similarities are marked as similar frames. Furthermore, similar frames are stored in a similar scene candidate queue, specifically stored sequentially according to cosine similarity.

[0101] It should be noted that, to improve the speed of calculating cosine similarity between the current starting frame and each image frame in the constructed image, and considering the characteristics of a parking lot scene, almost all image frames contain lane line semantics, while the quantity of lane line semantics has a relatively small impact on image similarity judgment; in contrast, elements such as speed bumps, sidewalks, and arrows are more important in scene recognition. Therefore, before calculating the cosine similarity between two bag-of-words model vectors, the magnitude of the two bag-of-words model vectors is first determined. If the magnitudes of the two vectors differ significantly, unnecessary calculations can be reduced, and the more discriminative semantics can be compared first, thereby improving overall computational efficiency.

[0102] Specifically, the process of calculating multiple cosine similarities based on the bag-of-words model vectors of each image frame and the bag-of-words model vector of the current starting frame is as follows (processes B1 to B3):

[0103] Process B1: For each image frame in each image frame, determine whether the bag-of-words model vector of the image frame is equal in size to the bag-of-words model vector of the current starting frame.

[0104] Understandably, for each image frame, it is determined whether the size of the bag-of-words model vector of that image frame is equal to that of the bag-of-words model vector of the current starting frame. If they are not equal, process B2 is executed; if they are equal, process B3 is executed.

[0105] Process B2: If they are not equal in size, remove the lane line semantics from the larger bag-of-words model vector and return to the step of determining whether the bag-of-words model vector of the image frame is equal in size to the bag-of-words model vector of the current starting frame.

[0106] It should be noted that if the size of the bag-of-words model vector of the image frame is not equal to that of the bag-of-words model vector of the current starting frame, the number of lane line semantics in the bag-of-words model vector with the larger size will be deleted until the size of the bag-of-words model vector of the image frame is equal to that of the bag-of-words model vector of the current starting frame.

[0107] It should be noted that if lane line semantics in a larger bag-of-words model vector have been completely removed, and the size of the bag-of-words model vector of the image frame is still not equal to that of the bag-of-words model vector of the current starting frame, then the deletion operation is stopped. Then, the image frame is skipped, and the cosine similarity between the bag-of-words model vector of that image frame and the bag-of-words model vector of the current starting frame is not calculated.

[0108] Process B3: If the sizes are equal, calculate the cosine similarity between the bag-of-words model vector of the image frame and the bag-of-words model vector of the current starting frame, and obtain multiple cosine similarities.

[0109] It is understandable that each image frame with a size equal to the bag-of-words model vector of the current starting frame is compared with the bag-of-words model vector of the current starting frame to calculate the cosine similarity, resulting in multiple cosine similarities.

[0110] It should be noted that the formula for calculating cosine similarity is shown in formula (2).

[0111] (2)

[0112] Where A and B are two bag-of-words model vectors whose similarity is being calculated; A×B is the dot product operation of bag-of-words model vector A and bag-of-words model vector B; Let A be the modulus of the bag-of-words model vector A; Let B be the modulus of the bag-of-words model vector.

[0113] It should be noted that the cosine similarity value ranges from -1 to +1. The closer the cosine similarity value is to 1, the more similar the two bag-of-words model vectors are.

[0114] For example, see the embodiments of the present invention. Figure 8 and Figure 9 As shown, Figure 8 and Figure 9 These are two frames taken during the driving process, one before the other, with a cosine similarity of 0.999035, indicating that the two images depict the same scene. Figure 8 and Figure 10 The two scenes shown are not similar, and the cosine similarity is 0.709268.

[0115] In particular, due to the presence of lane lines, the cosine similarity will not be very close to 0.

[0116] Step S104: Perform image-to-point-cloud operation on the semantic segmentation image of the current starting frame surround stitching image to obtain the target point cloud, and perform image-to-point-cloud operation on the semantic segmentation images of multiple similar frames to obtain multiple registered point clouds.

[0117] Understandably, after filtering out similar frames (i.e. similar scenes) through cosine similarity calculation, the semantic segmentation image of the current starting frame's around-view stitched image is used to perform image-to-point-cloud conversion with each map frame in the similar scene candidate queue.

[0118] It's important to note that the image-to-point-cloud conversion is achieved by transforming the image to the vehicle's coordinate system. This requires precise calibration of the extrinsic parameters between the camera and the vehicle beforehand. Accurate extrinsic parameter calibration ensures that each pixel in the image is correctly mapped to its position in the vehicle's coordinate system, thus generating accurate point cloud data.

[0119] Step S105: Perform point cloud registration between each registered point cloud and the target point cloud to obtain the pose matrix, and calculate the overlap between the current starting frame and multiple similar frames.

[0120] In the specific implementation step S105, each registered point cloud is registered with the target point cloud (that is, iterative closest point (ICP) registration) to obtain the pose matrix. The corresponding similar frames of the pose matrix are rotated and translated. Then, the overlap between the similar frames after rotation and translation and the current starting frame is calculated.

[0121] It should be noted that iterative nearest point registration is a commonly used point cloud registration algorithm. Through an iterative optimization process, it aligns one point cloud with another, minimizing the distance error between them.

[0122] Step S106: Determine the similar frame corresponding to the highest overlap among all overlap degrees as the final starting point, and use the pose matrix of the similar frame as the pose of the current starting frame in the mapping.

[0123] In the specific implementation step S105, the overlap between each similar frame and the current starting frame is sorted, the similar frame with the highest overlap is determined as the final starting point, and the pose matrix obtained by registering the similar frame is used as the pose of the current starting frame in the mapping.

[0124] In this embodiment of the invention, when the semantic richness of the current starting frame image meets the conditions, the perceptual reasoning result of the current starting frame surround-view stitched image is obtained, and cosine similarity is calculated with the perceptual result of the image frame in the constructed map to quickly match similar scenes, thus realizing arbitrary point localization in scenes that meet the conditions in the constructed map. This method can complete the starting point localization more quickly and flexibly, supports localization from any point in the map, has strong flexibility and adaptability, and significantly increases the application scope and robustness of the system.

[0125] Corresponding to the image matching method for assisted visual localization proposed in the above embodiments of the present invention, see [link to relevant documentation]. Figure 11 The diagram shows a structural block diagram of an image matching device for assisted visual positioning according to an embodiment of the present invention. The device includes: a calculation unit 1101, a conversion unit 1102, a selection unit 1103, an image-to-point-cloud unit 1104, a point-cloud registration unit 1105, and a determination unit 1106.

[0126] The calculation unit 1101 is used to calculate the image entropy of the current starting frame when the starting point positioning signal is received.

[0127] The transformation unit 1102 is used to transform the semantic segmentation image of the current starting frame surround stitching map into the bag-of-words model vector of the current starting frame if the image entropy is greater than a first threshold, and to transform the semantic segmentation image of each image frame in the mapping data into the bag-of-words model vector of each image frame.

[0128] The selection unit 1103 is used to select similar frames to the current starting frame from each image frame based on the bag-of-words model vector of each image frame and the bag-of-words model vector of the current starting frame.

[0129] The image-to-point-cloud unit 1104 is used to perform image-to-point-cloud operations on the semantic segmentation image of the current starting frame surround stitching image to obtain the target point cloud, and to perform image-to-point-cloud operations on the semantic segmentation images of multiple similar frames to obtain multiple registered point clouds.

[0130] The point cloud registration unit 1105 is used to register each registered point cloud with the target point cloud to obtain a pose matrix and calculate the overlap between the current starting frame and multiple similar frames.

[0131] The point cloud registration unit 1105 is specifically used for: rotating and translating the corresponding similar frames of the pose matrix; and calculating the overlap between the similar frames after rotation and translation and the current starting frame.

[0132] The determining unit 1106 is used to determine the similar frame corresponding to the highest overlap degree among all overlap degrees as the final starting point, and to use the pose matrix of the similar frame as the pose of the current starting frame in the mapping.

[0133] In this embodiment of the invention, when the semantic richness of the current starting frame image meets the conditions, the perceptual reasoning result of the current starting frame surround-view stitched image is obtained, and cosine similarity is calculated with the perceptual result of the image frame in the constructed map to quickly match similar scenes, thus realizing arbitrary point localization in scenes that meet the conditions in the constructed map. This method can complete the starting point localization more quickly and flexibly, supports localization from any point in the map, has strong flexibility and adaptability, and significantly increases the application scope and robustness of the system.

[0134] Combination Figure 11 The conversion unit 1102 shown includes: an expansion module, a downsampling module, an erosion module, and a storage module.

[0135] The dilation module is used to dilate each non-zero semantic element in the semantic segmentation image of the current starting frame lookaround stitching image.

[0136] The downsampling module is used to downsample the semantic segmentation image after dilation using an image resizing function.

[0137] The erosion module is used to perform an erosion operation on the downsampled semantic segmentation image to obtain the semantic segmentation image to be converted.

[0138] The storage module is used to convert the pixel value of each non-zero pixel in the semantic segmentation image to be transformed into a string type and store it in a hash table to obtain the bag-of-words model vector of the current starting frame.

[0139] Combination Figure 11 The content shown, selected unit 1103, includes: a calculation module and a marking module.

[0140] The calculation module is used to calculate multiple cosine similarities based on the bag-of-words model vectors of each image frame and the bag-of-words model vector of the current starting frame.

[0141] The labeling module is used to label image frames corresponding to multiple target cosine similarities as similar frames. The target cosine similarity is the cosine similarity among multiple cosine similarities that is greater than a second threshold.

[0142] Combination Figure 11 The content shown is a calculation module, which includes: a judgment submodule, a deletion submodule, and a calculation submodule.

[0143] The judgment submodule is used to determine whether the bag-of-words model vector of each image frame is equal in size to the bag-of-words model vector of the current starting frame for each image frame.

[0144] The deletion submodule is used to delete lane line semantics from the larger bag-of-words model vector if they are not equal in size, and then return to the execution judgment submodule.

[0145] The calculation submodule is used to calculate the cosine similarity between the bag-of-words model vector of the image frame and the bag-of-words model vector of the current starting frame if the sizes are equal, and obtain multiple cosine similarities.

[0146] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0147] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0148] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image matching method for assisting visual positioning, characterized by, The method comprises: When a starting point positioning signal is received, calculating image entropy of a current starting frame; If the image entropy is greater than a first threshold value, converting a semantic segmentation image of the current starting frame in a surround view stitching map into a bag-of-words vector of the current starting frame, and converting semantic segmentation images of each image frame in mapping data into bag-of-words vectors of each image frame; Based on the bag-of-words vectors of each image frame and the bag-of-words vector of the current starting frame, selecting similar frames of the current starting frame from each image frame; Performing image-to-point cloud operation on the semantic segmentation image of the current starting frame in the surround view stitching map to obtain a target point cloud, and performing image-to-point cloud operation on semantic segmentation images of a plurality of similar frames to obtain a plurality of registered point clouds; Performing point cloud registration on each registered point cloud and the target point cloud to obtain a pose matrix, and calculating a degree of coincidence between the current starting frame and the plurality of similar frames; Determining a similar frame corresponding to a highest degree of coincidence in the degrees of coincidence as a final starting point, and taking the pose matrix corresponding to the similar frame as a pose of the current starting frame in mapping.

2. The method of claim 1, wherein, The conversion of the semantic segmentation image of the current starting frame in the surround view stitching map into the bag-of-words vector of the current starting frame comprises: Performing dilation operation on each non-zero semantic in the semantic segmentation image of the current starting frame in the surround view stitching map; Performing down-sampling on the semantic segmentation image after the dilation operation by using an image size adjustment function; Performing erosion operation on the down-sampled semantic segmentation image to obtain a to-be-converted semantic segmentation image; Converting a pixel value of each non-zero pixel point in the to-be-converted semantic segmentation image into a string type, and storing the pixel value in a hash table to obtain the bag-of-words vector of the current starting frame.

3. The method of claim 1, wherein, The selection of the similar frames of the current starting frame from each image frame based on the bag-of-words vectors of each image frame and the bag-of-words vector of the current starting frame comprises: Based on the bag-of-words vectors of each image frame and the bag-of-words vector of the current starting frame, calculating a plurality of cosine similarities; Marking image frames corresponding to a plurality of target cosine similarities as similar frames, the target cosine similarities being cosine similarities greater than a second threshold value in the plurality of cosine similarities.

4. The method of claim 3, wherein, The calculation of the plurality of cosine similarities based on the bag-of-words vectors of each image frame and the bag-of-words vector of the current starting frame comprises: For each image frame in each image frame, judging whether the bag-of-words vector of the image frame and the bag-of-words vector of the current starting frame are equal in size; If not equal in size, deleting a number of lane line semantics in a larger bag-of-words vector, and returning to the step of judging whether the bag-of-words vector of the image frame and the bag-of-words vector of the current starting frame are equal in size; If equal in size, calculating a cosine similarity between the bag-of-words vector of the image frame and the bag-of-words vector of the current starting frame to obtain a plurality of cosine similarities.

5. The method of claim 1, wherein, The calculation of the degree of coincidence between the current starting frame and the plurality of similar frames comprises: Performing rotation and translation processing on the similar frames corresponding to the pose matrix; Calculating a degree of coincidence between the similar frames after the rotation and translation processing and the current starting frame.

6. An image matching device for assisting visual positioning, characterized by The device comprises: The computing unit is configured to calculate image entropy of a current starting frame when a starting point positioning signal is received; The converting unit is configured to convert the semantic segmentation image of the current starting frame into a bag-of-words vector of the current starting frame, and convert semantic segmentation images of each image frame in the mapping data into bag-of-words vectors of the image frames, if the image entropy is greater than a first threshold value; The selecting unit is configured to select a similar frame of the current starting frame from each image frame based on the bag-of-words vectors of the image frames and the bag-of-words vector of the current starting frame; The image-to-point cloud unit is configured to perform image-to-point cloud operation on the semantic segmentation image of the current starting frame of the surround view stitching map to obtain a target point cloud, and perform image-to-point cloud operation on semantic segmentation images of a plurality of similar frames to obtain a plurality of registered point clouds; The point cloud registration unit is configured to perform point cloud registration on each registered point cloud and the target point cloud to obtain a pose matrix, and calculate a degree of coincidence between the current starting frame and the plurality of similar frames; The determining unit is configured to determine a similar frame corresponding to a highest degree of coincidence among the degrees of coincidence as a final starting point, and determine a pose matrix corresponding to the similar frame as a pose of the current starting frame in the mapping.

7. The apparatus of claim 6, wherein, The converting unit comprises: An inflation module configured to perform inflation operation on each non-zero semantic in the semantic segmentation image of the current starting frame of the surround view stitching map; A down-sampling module configured to perform down-sampling on the semantic segmentation image after the inflation operation by using an image size adjustment function; An erosion module configured to perform erosion operation on the down-sampled semantic segmentation image to obtain a to-be-converted semantic segmentation image; A storage module configured to convert a pixel value of each non-zero pixel point in the to-be-converted semantic segmentation image into a string type, and store the pixel value in a hash table to obtain a bag-of-words vector of the current starting frame.

8. The apparatus of claim 6, wherein, The selecting unit comprises: A calculation module configured to calculate a plurality of cosine similarities based on the bag-of-words vectors of the image frames and the bag-of-words vector of the current starting frame; A marking module configured to mark image frames corresponding to target cosine similarities as similar frames, the target cosine similarities being cosine similarities greater than a second threshold value among the plurality of cosine similarities.

9. The apparatus of claim 8, wherein, The calculation module comprises: A judgment sub-module configured to judge whether a bag-of-words vector of each image frame among the image frames and the bag-of-words vector of the current starting frame are equal in size; A deletion sub-module configured to delete a number of lane line semantics in a larger bag-of-words vector if the bag-of-words vectors are not equal in size, and return to execute the judgment sub-module; A calculation sub-module configured to calculate a cosine similarity between the bag-of-words vector of the image frame and the bag-of-words vector of the current starting frame if the bag-of-words vectors are equal in size, to obtain a plurality of cosine similarities.

10. The apparatus of claim 6, wherein, The point cloud registration unit is specifically configured to: Perform rotation and translation processing on the similar frame corresponding to the pose matrix, and calculate a degree of coincidence between the similar frame after the rotation and translation processing and the current starting frame.

Images