Cross-modal data matching method and device for road video and map, and medium

Abstract

The invention relates to a cross-modal data matching method and device for a road video and a map and a medium, and the method comprises the following steps: obtaining point elements of the road video and the map, respectively extracting road indication arrows in the road video and arrow marking points in the point elements of the map, the arrow marking points being map points; matching the road indicating arrow with the arrow marking template, selecting a key point of the road indicating arrow, extracting a pixel coordinate of the key point, and converting the pixel coordinate of the key point into a geographic coordinate; by calculating the centroids of the key points and the centroids of the map points, aligning the main directions of the key points and the map points to obtain an alignment result; euclidean clustering is carried out on the key points and the map points respectively, structural information is calculated respectively, and data matching is realized by calculating the similarity of the structural information and optimizing the matching deviation of alignment results. Compared with the prior art, the method has the advantages of high accuracy, strong robustness and the like.

Description

Technical Field

[0001] The present invention relates to the field of navigation and positioning technology, and in particular to a method, device and medium for cross-modal data matching between road videos and maps. Background Art

[0002] Thanks to recent improvements in consumer drone performance, decreasing costs, and increasingly simple operation, drones are becoming accessible to ordinary consumers, enabling them to be widely used in aerial photography and entertainment. While fast, flexible, and cost-effective, consumer drones record limited sensor data, limited to location information at the time of capture. This limits their application in areas such as surveying and mapping and transportation.

[0003] If consumer-grade drone video data can be matched with high-precision maps and the video frames can be converted into raster data with geographic information close to the accuracy of high-precision maps, the application potential of consumer-grade drones in road infrastructure supervision, high-precision vehicle trajectory extraction, traffic flow characteristics analysis, driving behavior modeling, etc. can be greatly increased.

[0004] However, there is currently no method for cross-modal data matching between consumer drone videos and high-precision maps. Research has only focused on cross-modal matching between remote sensing imagery and vector maps. However, remote sensing images inherently have a certain degree of positional accuracy, resulting in relatively small scale and translation errors compared to the vector maps being matched. Videos captured by consumer drones, on the other hand, only have GPS location information at the time of capture. Even after positioning the video frames using these locations, significant scale, angle, and translation errors can still exist compared to high-precision maps. Therefore, a method for cross-modal data matching between consumer drone videos and high-precision maps is essential. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, device and medium for cross-modal data matching of road videos and maps in order to overcome the defect of large matching errors in the above-mentioned prior art.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A cross-modal data matching method for road videos and maps includes the following steps:

[0008] S1: Obtaining point features of the road video and the map, respectively extracting the road indication arrows in the road video and the arrow marking points in the point features of the map, where the arrow marking points are map points;

[0009] S2: Match the road sign arrow with a preset arrow marking template, select key points of the road sign arrow, extract the pixel coordinates of the key points, and convert the pixel coordinates of the key points into geographic coordinates;

[0010] S3: By calculating the center of mass of the key point and the center of mass of the arrow mark point, the key point and the main direction of the arrow mark point are aligned to obtain the alignment result;

[0011] S4: Perform Euclidean clustering on key points and arrow marking points along the road direction, and calculate the structural information of each cluster point cluster. By calculating the similarity of the structural information, select point clusters with high similarity to optimize the matching deviation of the alignment results and achieve data matching.

[0012] Furthermore, in step S1, adaptive density estimation and KNN clustering are performed on the road video to classify pixels in the video into foreground or background, retain the background portion to obtain a static image, and extract the road indication arrows in the video from the static image.

[0013] Furthermore, the extraction of road arrows in the video is specifically as follows: using the target detection network and the classification network to identify the arrow markings in the static image, obtain the arrow marking bounding box, and crop along the arrow marking bounding box to obtain the arrow marking image block.

[0014] Furthermore, step S2 includes the following steps:

[0015] S21: using a preset arrow marking template, which includes an image and pixel coordinates of key points of the arrow marking template, inputting the arrow marking image block and the arrow marking template into a feature matching model to obtain feature point matching pairs, solving the feature point matching pairs using a least squares method to obtain pixel coordinates of the key points;

[0016] S22: Calculate a coarse scale for converting pixel coordinates to geographic coordinates based on the actual length of the road indication arrow, and convert the pixel coordinates of the key point into geographic coordinates based on the coarse scale.

[0017] Furthermore, step S3 includes the following steps:

[0018] S31: Calculate the centroid of the key point and the map point. The calculation expression of the centroid is:

[0019]

[0020] Where, is the horizontal coordinate of the centroid of the key point, is the centroid ordinate of the key point, is the horizontal coordinate of the centroid of the map point, is the centroid ordinate of the map point, is the horizontal coordinate of the key point, is the vertical coordinate of the key point, is the horizontal coordinate of the map point, is the ordinate of the map point, l is the number of key points, and n is the number of map points;

[0021] S32: Calculate the covariance matrix of the key points and map points after removing the centroid. The calculation expression of the covariance matrix is:

[0022]

[0023] Where, is the horizontal coordinate of the key point after removing the centroid, is the vertical coordinate of the key point after removing the centroid, is the horizontal coordinate of the map point after removing the centroid, is the vertical coordinate of the map point after removing the centroid, C k is the covariance matrix of the key points, C m is the covariance matrix of the map points;

[0024] S33: Perform eigendecomposition on the covariance matrix to obtain the eigenvalue and eigenvector matrices. The calculation expression is:

[0025]

[0026] S34: Based on the eigenvector matrix, calculate the angle between the main direction of the key point and the main direction of the map point, and rotate the key point according to the angle to obtain the alignment result. The calculation expression of the rotation process is:

[0027]

[0028] Where α is the rotation angle, R is the rotation matrix, is the horizontal coordinate of the key point after alignment with the main direction, The vertical coordinate of the keypoint after alignment with the main direction.

[0029] Furthermore, step S4 includes the following steps:

[0030] For key points and map points, Euclidean clustering and screening are performed along the road direction with a fixed threshold to obtain multiple valid point clusters. The valid point clusters are connected along the direction perpendicular to the road to obtain key point segments and map point segments. Similarity calculation is performed based on the key point segments and map point segments, and segment pairs are matched to obtain the structural information of the segment pairs.

[0031] Furthermore, the structural information of the line segment pair includes the length of the key point line segment, the length of the map point line segment, the size of the minimum enclosing rectangle of the point cluster in the line segment, the scale information and the length of the connecting line between each adjacent node in the line segment.

[0032] Furthermore, in step S4, the similarity calculation is specifically as follows: by determining the feature similarity of the line segments, the matching line segments are determined, and the matching point pairs are further determined. The transformation parameters are calculated based on the matched point pairs to align the key points with the map points. Finally, the aligned key points are used as the initial values ​​of the scale-considered ICP point cloud registration algorithm for iterative optimization to obtain the transformation matrix that aligns the key points with the map points.

[0033] In a second aspect of the present invention, a device for cross-modal data matching between road videos and maps includes a memory, a processor, and a program stored in the memory. When the processor executes the program, it implements any of the above methods for cross-modal data matching between road videos and maps.

[0034] A third aspect of the present invention is a storage medium having a program stored thereon, which, when executed, implements any of the above-mentioned cross-modal data matching methods for road videos and maps.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1) This method uses video data, GPS positioning points at the time of capture, arrow marking templates, and high-precision maps of the area to achieve cross-modal data matching. Arrow markings are less likely to change in geometric position than other types of lane markings such as solid lines and dashed lines. Therefore, arrow markings are used as a key element in matching data from two different modalities: video and map. Through marking matching, centroid calculation, and Euclidean clustering, the angular differences, size, and translation deviations between key points in the video and map points are gradually eliminated to obtain accurate positioning results.

[0037] 2) The results of the present invention are highly accurate and robust, and can provide reliable support for the application of consumer-grade drone video data in areas such as road facility inspection and maintenance, traffic monitoring and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of the present invention.

[0039] Figure 2 This is the data pre-processing flow chart.

[0040] Figure 3 This is the arrow marking identification effect diagram.

[0041] Figure 4 Extract schematic diagram for key points.

[0042] Figure 5 Schematic diagram of the preliminary conversion of key point pixel coordinates into geographic coordinates.

[0043] Figure 6 This is the initial transformation effect diagram of key point coordinates.

[0044] Figure 7 This is the effect diagram of aligning the main directions of key points and map points.

[0045] Figure 8 Schematic diagram of the key point clustering and structural information calculation process.

[0046] Figure 9 The effect diagram of optimizing the matching between key points and map points.

[0047] Figure 10 This is the cross-modal data matching result between consumer drone videos and high-precision maps. DETAILED DESCRIPTION

[0048] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0049] Example 1

[0050] The present invention provides a method, device and medium for cross-modal data matching between road videos and maps. Figure 1 As shown, the method includes the following steps:

[0051] S1: Obtaining point features of the road video and the map, respectively extracting the road indication arrows in the road video and the arrow marking points in the point features of the map, where the arrow marking points are map points;

[0052] Arrow recognition involves obtaining the bounding boxes and category information of arrow markings from low-cost drone video data. This involves two steps.

[0053] S11: Video data pre-processing

[0054] Video data pre-processing process is as follows Figure 2 To reduce the occlusion of arrow markings by dynamic objects such as cars and pedestrians, we perform efficient adaptive density estimation on each pixel in the video data frame, then combine it with KNN clustering to classify the pixels as foreground or background. The background pixels are retained to obtain a clean image frame without dynamic objects.

[0055] S12: Traditional feature engineering methods require manual design of the geometric and attribute features of arrow markings. This is not flexible enough in the face of complex road environments and lighting conditions, and cannot effectively identify arrow markings. Therefore, a deep learning-based method is used to combine the target detection network PP-YOLOE+ with the classification network PP-HGNetV2 to recognize arrow markings in video frames. The recognition effect is as follows: Figure 3 As shown, Figure 3The white box in the middle represents the recognized road arrow markings.

[0056] S2: Match the road sign arrow with a preset arrow marking template, select key points of the road sign arrow, extract the pixel coordinates of the key points, and convert the pixel coordinates of the key points into geographic coordinates;

[0057] Arrow markings in HD maps are represented as point-like vector features, with the geographic location of the point at the center of the arrow's tail. To match the arrow markings in the video frame with those in the HD map, the pixel coordinates of the corresponding key points are extracted based on their bounding boxes and corresponding categories, and the pixel coordinates are initially converted to geographic coordinates. This step consists of two steps: key point extraction and preliminary key point coordinate conversion.

[0058] S21: using a preset arrow marking template, which includes an image and pixel coordinates of key points, inputting the arrow marking image block and the arrow marking template into a feature matching model to obtain feature point matching pairs, solving the feature point matching pairs using a least squares method to obtain pixel coordinates of the key points;

[0059] The key point extraction method based on arrow mark template feature matching is used. The specific process is summarized in Figure 4 First, prepare the arrow marking template. Figure 4 Part (a) is a schematic diagram of the template, which includes the image and the pixel coordinates of the key points. Based on the arrow mark bounding box obtained in the previous step, the arrow mark image block is cropped from the clean frame. Then, the corresponding arrow mark template is selected according to the category of the arrow mark, and the arrow mark image block and the arrow mark template are input into the LightGlue feature matching model to obtain n feature point matching pairs {(p i ,t i )} i＝0,…,n ,like Figure 4 As shown in part (b) of i is the feature point of the arrow-marked image block, t i For the feature points of the corresponding template. For n matching point pairs, the least squares method is used to solve the homography matrix H so that for each point t i ,have:

[0060] p i ≈Ht i

[0061] After solving the homography matrix H, the key point coordinates of the template are Convert to image blocks by:

[0062]

[0063] Get the pixel coordinates of the key points of the arrow mark image block Complete key point extraction.

[0064] S22: Calculate a coarse scale for converting pixel coordinates to geographic coordinates based on the actual length of the road indication arrow, and convert the pixel coordinates of the key point into geographic coordinates based on the coarse scale.

[0065] After obtaining the pixel coordinates of the key points, they need to be converted to geographic coordinates. Since consumer drone video data only has GPS information at the time of acquisition, without information such as pitch angle and yaw angle, and there are no pre-set control points on the ground, it is necessary to combine the national standards for road markings to achieve the initial conversion of the geographic coordinates of the key points. The steps are as follows: Figure 5 shown.

[0066] Get the coordinates of the positioning point (Geo x ,Geo y ), and then assume that the center point of the video frame (C x ,C y ) is the coordinate of the positioning point. Figure 5 Taking the straight arrow in the figure as an example, the pixel length l of the arrow marking has been obtained in the above steps. By querying the national standard for road markings, the length of the straight arrow corresponding to this road section is a meters. The coarse scale of the pixel coordinate conversion to geographic coordinates can be calculated by the following formula:

[0067]

[0068] After obtaining the coarse scale, the coarse geographic coordinates of the origin of the upper left corner of the video frame are calculated by the following formula:

[0069]

[0070] Based on the rough geographic coordinates of the upper left corner of the video, the pixel coordinates of the key points extracted in the previous step (P x ,P y ) can be converted into geographic coordinates by the following formula

[0071]

[0072] like Figure 6 The figure shows the result of the preliminary coordinate transformation of the key points. It can be seen that there are still deviations in scale, angle, and distance between the key points in this area and the map points (the locations of the arrow lines in the high-precision map).

[0073] S3: By calculating the center of mass of the key point and the center of mass of the arrow mark point, the key point and the main direction of the arrow mark point are aligned to obtain the alignment result;

[0074] One of the applications of this invention is the positioning application of consumer drones. Due to the limited sensor information provided by consumer drones, the geographic information of key points calculated based solely on GPS positioning points has a large positional deviation from the arrow marking points on the high-precision map. In addition, the points are very sparse. It is difficult for feature-based or learning-based point cloud registration methods to obtain point features to achieve matching between key points and map points. The key point and map point optimization matching method proposed in this patent has the following steps:

[0075] S31: Calculate the centroid of key points and map points;

[0076] by Represents a key point set of l points, Representing a map point set of n points, the centroid of the key points is calculated by the following formula Centroid of the map point

[0077]

[0078] After obtaining the centroid, the coordinates of the key points after removing the centroid With map point coordinates Calculated by the following formula:

[0079]

[0080] S32: Calculate the covariance matrix of the key points and map points after removing the centroid;

[0081] Calculate the covariance matrix C of the key points and map points after removing the centroid k and C m , the calculation expression of the covariance matrix is:

[0082]

[0083] S33: Perform eigendecomposition on the covariance matrix to obtain the eigenvalue and eigenvector matrices. The calculation expression of the eigendecomposition is:

[0084]

[0085] S34: Based on the eigenvector matrix, calculate the angle between the main direction of the key point and the main direction of the map point, and rotate the key point according to the angle to obtain the alignment result:

[0086]

[0087] Construct the rotation matrix R by the rotation angle, rotate the key point and add the centroid of the map point set to obtain the key point coordinates after the main direction is aligned The calculation expressions of the rotation matrix R and the rotation process are:

[0088]

[0089] The key points after the main direction alignment are as follows Figure 7 As shown by the black dot in , it can be seen that the angle deviation from the map point is reduced.

[0090] S4: Perform Euclidean clustering on key points and arrow marking points along the road direction, and calculate the structural information of each cluster point cluster. By calculating the similarity of the structural information, select point clusters with high similarity to optimize the matching deviation of the alignment results and achieve data matching.

[0091] After aligning the main direction of the key point with the main direction of the map point, the angle deviation between the two is reduced, but there is still a deviation in scale and displacement. To further match, construct line segments through key points and extract the structural information corresponding to the line segments as features. The specific steps are as follows: Figure 8 As shown. Since the key point is the midpoint of the tail of the arrow mark, the main direction of the key point set is horizontal to the direction of the road after being aligned with the main direction of the map point. The road direction is Figure 8 Along the direction of the road, with a fixed distance threshold c, the key points and map points are clustered separately in an Euclidean manner, and clusters with less than 2 points are removed, such as Figure 8 As shown in part (b) of . Each cluster of points obtained by clustering is basically the tail point of the arrow mark that is close in the direction perpendicular to the road, so it will be along the direction perpendicular to the road, that is, Figure 8 In the W-axis direction of part (a), each point cluster is connected into a line segment. In order to calculate the similarity between line segments to obtain matching line segment pairs, the structural information of the line segments is extracted as features, such as Figure 8 As shown in part (c) of the figure, the structural information includes the length of the line connecting its starting and ending points as the length of the line segment. A minimum enclosing rectangle is constructed for each point cluster in the line segment, and the width, height, and aspect ratio of the rectangle are used as the width, height, and scale information of the line segment, respectively. Furthermore, the length of the connecting line between each adjacent node in the line segment also serves as its structural information.

[0092] The structural information of the line segments constructed by the obtained point clusters is used to calculate the similarity between the line segments. The matching line segments are determined by the feature similarity of different line segments, and the matching point pairs are further obtained from the paired line segments. The transformation parameters are then calculated based on the matched point pairs to align the key points with the map points. Finally, the aligned key points are used as the initial values ​​of the scale-considered ICP point cloud registration algorithm for iterative optimization to obtain the transformation matrix that aligns the key points with the map points. The result after optimized matching of key points and map points is as follows: Figure 9 As shown in Figure 2, we can see that the optimized matching method between key points and map points effectively reduces the deviations in angle, scale, and translation between the two.

[0093] The coordinate conversion parameters obtained through the above steps can be used to convert the pixel coordinates of any point in the video captured by the consumer drone to the geographic coordinate system used by the high-precision map, that is, to convert the video frame into a raster image with an accuracy close to that of the high-precision map. Figure 10 As shown in the figure, it can be seen that this method effectively completes the cross-modal data matching between video and high-precision map.

[0094] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0095] In the context of the present invention, machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0096] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A cross-modal data matching method for road videos and maps, characterized in that: The following steps are involved: S1: Obtaining point elements of a road video and a map, respectively extracting road indication arrows in the road video and arrow marking points in the point elements of the map, wherein the arrow marking points are map points; S2: Match the road sign arrow with a preset arrow marking template, select key points of the road sign arrow, extract the pixel coordinates of the key points, and convert the pixel coordinates of the key points into geographic coordinates; S3: By calculating the center of mass of the key point and the center of mass of the arrow mark point, the key point and the main direction of the arrow mark point are aligned to obtain the alignment result; S4: Perform Euclidean clustering on key points and arrow marking points along the road direction, and calculate the structural information of each cluster point cluster. By calculating the similarity of the structural information, select point clusters with high similarity to optimize the matching deviation of the alignment results and achieve data matching.

2. A cross-modal data matching method for road videos and maps according to claim 1, characterized in that: In step S1, adaptive density estimation and KNN clustering are performed on the road video to classify the pixels in the video into foreground or background, and the background portion is retained to obtain a static image. The road arrows in the video are extracted from the static image.

3. The cross-modal data matching method for road videos and maps according to claim 2, characterized in that: The method of extracting road arrows from a video specifically includes: using a target detection network and a classification network to identify arrow markings in a static image, obtaining an arrow marking boundary box, and cropping along the arrow marking boundary box to obtain an arrow marking image block.

4. The cross-modal data matching method for road videos and maps according to claim 3, characterized in that: Step S2 includes the following steps: S21: using a preset arrow marking template, the arrow marking template including an image and pixel coordinates of key points of the arrow marking template, inputting the arrow marking image block and the arrow marking template into a feature matching model to obtain feature point matching pairs, solving the feature point matching pairs using a least squares method to obtain pixel coordinates of the key points; S22: Calculate a coarse scale for converting pixel coordinates to geographic coordinates based on the actual length of the road indication arrow, and convert the pixel coordinates of the key point into geographic coordinates based on the coarse scale.

5. The cross-modal data matching method for road videos and maps according to claim 1, characterized in that: Step S3 includes the following steps: S31: Calculate the centroid of the key point and the map point. The calculation expression of the centroid is: Where, is the horizontal coordinate of the center of mass of the key point, is the centroid ordinate of the key point, is the horizontal coordinate of the centroid of the map point, is the centroid ordinate of the map point, is the horizontal coordinate of the key point, is the vertical coordinate of the key point, is the horizontal coordinate of the map point, is the ordinate of the map point, l is the number of key points, and n is the number of map points; S32: Calculate the covariance matrix of the key points and map points after removing the centroid. The calculation expression of the covariance matrix is: Where, is the horizontal coordinate of the key point after removing the centroid, is the vertical coordinate of the key point after removing the centroid, is the horizontal coordinate of the map point after removing the centroid, is the vertical coordinate of the map point after removing the centroid, C k is the covariance matrix of the key points, C m is the covariance matrix of map points; S33: Perform eigendecomposition on the covariance matrix to obtain the eigenvalue and eigenvector matrices. The calculation expression is: S34: Based on the eigenvector matrix, calculate the angle between the main direction of the key point and the main direction of the map point, and rotate the key point according to the angle to obtain the alignment result. The calculation expression of the rotation process is: Where α is the rotation angle, R is the rotation matrix, is the horizontal coordinate of the key point after alignment with the main direction, The vertical coordinate of the keypoint after alignment with the main direction.

6. The cross-modal data matching method for road videos and maps according to claim 1, characterized in that: Step S4 includes the following steps: For key points and map points, Euclidean clustering and screening are performed along the road direction with a fixed threshold to obtain multiple valid point clusters. The valid point clusters are connected along the direction perpendicular to the road to obtain key point segments and map point segments. Similarity calculation is performed based on the key point segments and map point segments, and segment pairs are matched to obtain the structural information of the segment pairs.

7. The cross-modal data matching method for road videos and maps according to claim 6, characterized in that: The structural information of the line segment pair includes the length of the key point line segment, the length of the map point line segment, the size of the minimum enclosing rectangle of the point cluster in the line segment, the scale information and the length of the connecting line between each adjacent node in the line segment.

8. The cross-modal data matching method for road videos and maps according to claim 1, characterized in that: In step S4, the similarity calculation is specifically as follows: determining the matching line segments through the feature similarity of the line segments, further determining the matching point pairs, calculating the transformation parameters based on the matched point pairs to align the key points with the map points, and finally using the aligned key points as the initial values ​​of the scale-sensitive ICP point cloud registration algorithm for iterative optimization to obtain the transformation matrix that aligns the key points with the map points.

9. A cross-modal data matching device for road videos and maps, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, it implements a cross-modal data matching method for road video and map as described in any one of claims 1-8.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, a cross-modal data matching method for road video and map is implemented as described in any one of claims 1-8.

Images