Tile icon scaling method and device based on image registration and storage medium

Abstract

The invention is suitable for the field of image recognition, and provides a tile icon calibration method and device based on image registration and a storage medium, the tile icon calibration method based on image registration comprises the steps that a high-precision map of a preset road section and a shot image of a roadside camera are acquired, the high-precision map comprises tile map units of multiple levels, and the high-precision map comprises tile map units of multiple levels; the adjacent high-level tile map units comprise a plurality of corresponding low-level tile map units. And generating a tile mask graph according to the longitude and latitude of the base station and the tile graph unit of each level. And calibrating external parameters of the roadside camera according to the tile mask graph and the shot image. According to the scheme, the problems that time and labor are consumed and accuracy is limited when the feature mapping points of the image and the high-precision map are obtained manually are solved.

Description

Technical Field

[0001] The present application belongs to the field of image recognition, and in particular, relates to a method, device and storage medium for tile image localization based on image registration. Background Art

[0002] Image perception technology is widely used in roadside systems and is often used in traffic monitoring, road condition monitoring, and traffic signal control. In general, applying image perception technology to roadside systems can help improve the intelligence level of traffic management, enhance traffic safety and efficiency, and improve the urban traffic environment.

[0003] Although image detection algorithms are widely used in the fields of image processing and computer vision, they still have some shortcomings. For example, image detection algorithms can only provide 2D perception information on image coordinates, which is difficult to directly integrate with the 3D information obtained by LiDAR. In order to obtain the location information of the target in the earth coordinate system, it is necessary to use high-precision maps to assist image detection algorithms. The traditional conversion relationship between images and earth coordinate systems requires manual acquisition of feature mapping points of images and high-precision maps, and calculation of mapping matrices based on these feature points. However, this process is time-consuming and labor-intensive and has limited accuracy. Therefore, it is crucial to automatically obtain the mapping relationship from image coordinate system to earth coordinate system and realize accurate conversion of image coordinates to earth coordinates. Summary of the invention

[0004] The purpose of this application is to provide a method, apparatus, terminal device and storage medium for tile map localization based on image registration, aiming to solve the problem of how to automatically obtain the mapping relationship between the image coordinate system and the earth coordinate system.

[0005] A first aspect of an embodiment of the present application provides a method for tile image localization based on image registration, the method comprising:

[0006] Acquire a high-precision map of a preset road section and an image captured by a roadside camera, wherein the high-precision map includes multiple levels of tile image units, and an adjacent higher-level tile image unit includes a corresponding plurality of lower-level tile image units;

[0007] Generate a tile mask map according to the latitude and longitude of the base station and the tile map units of each layer;

[0008] The extrinsic parameters of the roadside camera are calibrated according to the tile mask image and the captured image.

[0009] In an embodiment that can be implemented in this application, the step of generating the high-precision map includes:

[0010] Projecting the surface points of the preset road section into a plane map by Mercator projection;

[0011] Divide the planar map into multiple levels of tile map units to generate the high-precision map.

[0012] In an implementable embodiment of the present application, a tile mask map is generated according to the base station longitude and latitude and each level of tile map unit, including:

[0013] Obtain the tile map units within a set range according to the base station longitude and latitude and the base station sensing distance;

[0014] Stitch the filtered tile maps to obtain a high-precision map within a set range;

[0015] Erase the non-concerned areas in the stitched high-precision map to generate a tile map mask map.

[0016] In an implementable embodiment of the present application, calibrating the external parameters of the roadside camera according to the tile mask map and the captured image includes:

[0017] Perform image registration on the tile mask map and the captured image to determine the coordinate mapping relationship between the tile mask map and the captured image;

[0018] Calibrate the external parameters of the roadside camera according to the coordinate mapping relationship between the tile mask map and the captured image.

[0019] In an implementable embodiment of the present application, performing image registration on the tile mask map and the captured image to determine the coordinate mapping relationship between the tile mask map and the captured image includes:

[0020] Calculate the key feature points of the tile mask map and the captured image;

[0021] Match the key feature points of the tile mask map and the captured image to obtain multiple key feature point matching pairs;

[0022] Calculate the coordinate mapping relationship between the tile mask map and the captured image according to all the key feature point matching pairs.

[0023] In an implementable embodiment of the present application, calculating the key feature points of the tile mask map and the captured image includes:

[0024] Construct Gaussian pyramids of the tile mask map and the captured image;

[0025] Generate a difference pyramid of the tile mask map according to the tile mask map, and generate a difference pyramid of the captured image according to the Gaussian pyramid of the captured image;

[0026] Find local extreme points in each difference pyramid, and use the local extreme points not less than the set extreme threshold as the key feature points;

[0027] Perform sub-pixel localization on the key feature points using Taylor expansion to determine the coordinates of each key feature point;

[0028] For each key feature point, calculate the gradient histogram in the area around the key feature point, obtain the histogram peak as the main gradient direction, and align the calculation direction of the feature descriptor to the main gradient direction.

[0029] In an implementable embodiment of the present application, the matching of the key feature points of the tile mask map and the captured image to obtain multiple key feature point matching pairs includes:

[0030] Determine multiple key feature point matching pairs based on the feature distances between the key feature points of the tile mask map and the captured image.

[0031] In an implementable embodiment of the present application, the method further includes:

[0032] Use interpolation mean to fill in the missing values in the captured image, find the nearest points by traversing in a top-left to bottom-right cycle, and perform interpolation averaging on the found left-right pairs or up-down pairs of points;

[0033] If no paired points with three-dimensional position information are found after the top-left to bottom-right search, assign the nearest non-empty value in the vicinity to this point.

[0034] A second aspect of the embodiments of the present application provides a device for calibrating tile icons based on image registration, and the calibration device includes:

[0035] An acquisition module that acquires a high-precision map of a preset road section and a captured image of a roadside camera, where the high-precision map includes multiple levels of tile map units, and adjacent higher-level tile map units include corresponding multiple lower-level tile map units;

[0036] A tile mask map generation module that generates a tile mask map according to the base station longitude and latitude and each level of tile map unit;

[0037] A calibration module that calibrates the external parameters of the roadside camera according to the tile mask map and the captured image.

[0038] A third aspect of the embodiments of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0039] Beneficial effects of this application

[0040] By adopting a tile icon calibration method based on image registration provided by the present application, the calibration of the camera is achieved by obtaining the tile map of a preset road and the captured photos and the matching pairs generated by their matching, and finally the position information of the target in the earth coordinate system is obtained, solving the problem that it is time-consuming, laborious and inaccurate to manually obtain the mapping relationship from the image coordinate system to the earth coordinate system. Description of the Drawings

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0042] Figure 1 It is a flowchart of a tile icon calibration method based on image registration provided by an embodiment of the present application;

[0043] Figure 2 It is a flowchart of the steps for generating a high-precision map provided by an embodiment of the present application;

[0044] Figure 3 It is a result diagram of a tile map of level 23 obtained by Mercator projection of a partial road end of a four-lane in a real road scene provided by an embodiment of the present application;

[0045] Figure 4 It is a flowchart of generating a tile mask map according to the base station longitude and latitude and the tile map units of each level provided by an embodiment of the present application;

[0046] Figure 5 It is a flowchart of calibrating the external parameters of the roadside camera according to the tile mask map and the captured image provided by an embodiment of the present application;

[0047] Figure 6 It is a flowchart of calibrating the external parameters of the roadside camera according to the coordinate mapping relationship between the tile mask map and the captured image provided by an embodiment of the present application;

[0048] Figure 7 It is a flowchart of calculating the key feature points of the tile mask map and the captured image provided by an embodiment of the present application;

[0049] Figure 8 It is a flowchart of filling in the missing values by the calibration method provided by an embodiment of the present application;

[0050] Figure 9 It is a captured image with missing pixel values provided by an embodiment of the present application;

[0051] Figure 10 The captured image after filling pixel values provided by the embodiments of this application;

[0052] Figure 11 The structural schematic diagram of a device for calibrating tile icons based on image registration provided by the embodiments of this application;

[0053] Figure 12 The structural schematic diagram of an electronic device provided by the embodiments of this application. Detailed implementation manners

[0054] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of this application. However, those skilled in the art should clearly understand that this application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of this application.

[0055] It should be understood that when used in the specification of this application and the appended claims, the term "including" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0056] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0057] As used in the specification of this application and the appended claims, the term "if" can be interpreted as "when", "once", "in response to determining", or "in response to detecting" according to the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" according to the context.

[0058] In addition, in the description of the specification of this application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0059] References to "one embodiment" or "some embodiments" etc. described in the specification of the present application mean that specific features, structures or characteristics described in connection with that embodiment are included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" etc. that appear at different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprise", "include", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0060] It should be understood that the magnitude of the sequence numbers of the steps in this embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0061] The application background of image detection algorithms is very extensive, involving multiple fields such as transportation, security, medical, industry, retail, agriculture, etc., providing powerful technical support for all walks of life. In the transportation field, the application of image detection algorithms is also relatively extensive at present, involving multiple aspects such as traffic monitoring and management, intelligent traffic signal control, autonomous driving technology, vehicle identification and tracking, intelligent traffic safety, etc., providing important support for the intelligent, informatized and secure development of the transportation industry. However, at present, in order to obtain the position information of the target in the earth coordinate system, it is necessary to rely on a high-precision map to assist the image detection algorithm. The traditional conversion relationship between the image and the earth coordinate system requires manually obtaining the feature mapping points of the image and the high-precision map, and calculating the mapping matrix based on these feature points. Manual calculation is not only cumbersome but also may have a large error. Therefore, this solution provides a method for calibrating tile icons based on image registration to solve this problem. By obtaining the tile map of a preset road and the taken photos and the matching pairs generated by their matching, the calibration of the camera is realized, and finally the position information of the target in the earth coordinate system is obtained.

[0062] In order to illustrate the technical solution of the present application, the following will be described through specific embodiments.

[0063] Refer to Figure 1 , which shows a flowchart of a method for calibrating tile icons based on image registration provided by an embodiment of the present application.

[0064] S101, obtain a high-precision map of a preset road section and the captured image of a roadside camera, the high-precision map includes multiple levels of tile map units, and adjacent higher-level tile map units include corresponding multiple lower-level tile map units.

[0065] S102. Generate a tile mask map based on the base station's longitude and latitude and the tile map units of each level.

[0066] S103. Calibrate the external parameters of the roadside camera according to the tile mask map and the captured image.

[0067] Exemplarily, assume that there is an existing road captured image of a preset section by a roadside camera and a high-precision map of the preset section. A tile mask map is generated through the base station's longitude and latitude and the tile map units of each level. Finally, the key feature points in the captured image are matched with the key feature points at the corresponding positions in the tile mask map to determine the external parameters of the camera, and ultimately obtain the position information of the target in the earth coordinate system.

[0068] The method of this application can automatically obtain the mapping relationship from the image coordinate system to the earth coordinate system, not only realizing the accurate positioning and matching of the roadside camera image and the map, but also improving the efficiency of obtaining the mapping relationship.

[0069] As Figure 2 shown, in the implementable embodiments of this application, the generation steps of the high-precision map include:

[0070] S201. Project the surface points of the preset section onto a planar map through Mercator projection.

[0071] S202. Divide the planar map into tile map units of multiple levels to generate the high-precision map.

[0072] Among them, the tile map projects the earth's surface points onto a plane through Mercator projection, and then divides the world map into map units of 256×256 pixels by means of cutting. Specifically, the tile map level and the tile map coordinate encoding are unique and the resolution is fixed at 256×256.

[0073] When the minimum map level is 0, the higher the tile map level, the more tile numbers that make up the world map, and the higher the map accuracy.

[0074] The tiles of any tile level map are composed of 4 tiles cut from each tile map of the lower level, forming a tile pyramid.

[0075] The upper left corner of the tile map is the origin of the tile map coordinate system. The positive direction of the X-axis is consistent with the positive direction of longitude and is to the left, and the positive direction of the Y-axis is consistent with the positive direction of latitude and is upward. Figure 3 The result map shows the partial road end of four lanes in the real road scene projected through Mercator projection to obtain a tile map of level 23.

[0076] Exemplarily, the process of projecting the surface points of the preset road section onto a planar map through Mercator projection includes: determining the projection center. Since Mercator projection is a cylindrical projection, it is necessary to determine the central longitude of the projection. Usually, the central longitude of Mercator projection is set to 0 degrees, that is, it is located on the prime meridian of the earth. Calculating the projection coordinates. For a given point (longitude, latitude) on the earth's surface, the Mercator projection coordinates (x, y) can be calculated using the following formula: where R is the radius of the earth, λ is the longitude, and φ is the latitude. Converting the projection coordinates to planar coordinates. By appropriately scaling and translating the calculated Mercator projection coordinates, they can be mapped to the actual coordinate system on the plane, thereby obtaining the final planar coordinates. It should be noted that Mercator projection is an equiangular projection, which can keep the angles unchanged but cannot keep the area and shape. Finally, the planar map obtained through Mercator projection is divided into multiple levels of tile map units to generate the high-precision map.

[0077] This step of the present application accurately obtains the high-precision map of a certain preset road section, improving the accuracy of the calibration result.

[0078] The tile mask map is shown in Figure 4 , in the implementable embodiments of the present application, S102 specifically includes:

[0079] Obtaining tile map units within a set range according to the longitude and latitude of the base station and the sensing distance of the base station;

[0080] Stitching the filtered tile maps to obtain a high-precision map within a set range;

[0081] Erasing the non-concerned areas in the stitched high-precision map to generate a tile map mask map.

[0082] As Figure 5 shown, in the implementable embodiments of the present application, S103 specifically includes:

[0083] S501, performing image registration on the tile mask map and the captured image to determine the coordinate mapping relationship between the tile mask map and the captured image;

[0084] S502, calibrating the external parameters of the roadside camera according to the coordinate mapping relationship between the tile mask map and the captured image.

[0085] Among them, when the image detection model detects a target, the coordinate information of the captured image is converted into the coordinate information on the corresponding tile map. The longitude and latitude in the tile map can be converted with the tile coordinates and the coordinates between the tile pixel points. The conversion relationship between the longitude and latitude coordinates and the tile map coordinates is as follows:

[0086]

[0087]

[0088] Among them, Lon and Lat respectively represent the longitude and latitude coordinate values, Level represents the tile level, and tileX and tileY respectively represent the tile coordinate values. The conversion relationship between the longitude and latitude coordinates and the tile image pixel coordinate system is as follows:

[0089]

[0090]

[0091] Among them, pixelX and pixelY respectively represent the tile pixel coordinate values. The conversion relationship between the tile pixel coordinates and the longitude and latitude coordinates is as follows:

[0092]

[0093]

[0094] Exemplarily, assume that there is a longitude and latitude coordinate of (40.7128°N, 74.0060°W), and convert it to the tile map coordinates. At this time, use the zoom level Level = 12 for conversion, converting the longitude and latitude coordinates to tile map coordinates:

[0095] According to the above conversion formula, the tile map coordinates corresponding to the longitude and latitude coordinates of New York City can be calculated:

[0096] lon = 74.0060 lat = 40.7128 Level = 12

[0097] tileX = ((lon + 180) / 360) * pow(2, Level) = ((74.0060 + 180) / 360) * pow(2, 12) = 916.0478

[0098] tileY = (1 - log(tan(lat * pi / 180) + 1 / cos(lat * pi / 180)) / pi) / 2 * pow(2, Level) = (1 - log(tan(40.7128 * pi / 180) + 1 / cos(40.7128 * pi / 180)) / pi) / 2 * pow(2, 12) = 2065.7994

[0099] Therefore, the tile map coordinates corresponding to the longitude and latitude coordinates (40.7128°N, 74.0060°W) at the zoom level of 12 are (916.0478, 2065.7994).

[0100] The process of converting tile map coordinates to longitude and latitude coordinates is as follows:

[0101] If there is a tile map coordinate (916.0478, 2065.7994) and a zoom level Level = 12, the inverse transformation can be performed using the formula for converting tile map coordinates to longitude and latitude coordinates to obtain the corresponding longitude and latitude coordinates of (40.7128°N, 74.0060°W). Extract each key feature point in the image, form multiple key feature point pairs based on the tile mask map and multiple key feature points corresponding to the captured image, and determine the mapping relationship between the coordinates of the tile mask map and the captured image according to the conversion relationship between the coordinates of multiple key feature point pairs. According to this mapping relationship, the external parameters of the roadside camera can be calibrated.

[0102] This method of the present application determines the external parameters of the roadside camera by obtaining the mapping relationship between the coordinates of the tile mask map and the captured image, thereby being able to accurately measure and analyze the objects and scenes in the captured image, and realizing applications such as traffic monitoring, traffic safety analysis, and traffic flow statistics. At the same time, calibrating the external parameters of the roadside camera can also eliminate image distortion and errors, improve the image quality and accuracy, and further improve the accuracy and reliability of traffic monitoring and analysis.

[0103] As Figure 6 shown, in the implementable embodiments of the present application, the image registration of the tile mask map and the captured image in S502 to determine the coordinate mapping relationship between the tile mask map and the captured image includes:

[0104] S601, calculating the key feature points of the tile mask map and the captured image;

[0105] S602, matching the key feature points of the tile mask map and the captured image to obtain multiple key feature point matching pairs;

[0106] S603, calculating the coordinate mapping relationship between the tile mask map and the captured image according to all key feature point matching pairs.

[0107] Exemplarily, first import the corresponding image processing library, then use the SIFT (Scale-Invariant Feature Transform) algorithm to extract the key feature points of the tile mask map and the captured image, use the feature point matching algorithm to match the key feature points in the extracted tile mask map and the captured image to obtain multiple matching pairs of key feature points, and generate the coordinate mapping relationship between the tile mask map and the captured image according to these matching pairs of key feature points.

[0108] Specifically, import the tile mask map and the captured photo of a preset road section into the image processing library at the same time. Use the SIFT algorithm to extract the feature points such as street lights, road signs, traffic lights, and pedestrians in the tile mask map, and regard these feature points as the key feature points of the tile mask map. At the same time, extract the feature points such as street lights, road signs, traffic lights, and pedestrians in the captured image, and regard these feature points as the key feature points of the captured image. Then use the feature point matching algorithm to match the key feature points in the two images to be matched, generate multiple key feature point matching pairs, and calculate the coordinate mapping relationship between the two images to be matched according to the coordinate relationship between multiple pairs of matching pairs. The two images to be matched mentioned above are the tile mask map and the captured image.

[0109] This step of the present application can quickly obtain the coordinate mapping relationship between the two images to be matched, realize the correspondence between the two images to be matched, and provide a certain basis for realizing the calibration of the camera.

[0110] As Figure 7 shown, in an implementable embodiment of the present application, calculating the key feature points of the tile mask map and the captured image in S601 includes:

[0111] S701, constructing Gaussian pyramids of the tile mask map and the captured image;

[0112] S702, generating a difference pyramid of the tile mask map according to the tile mask map, and generating a difference pyramid of the captured image according to the Gaussian pyramid of the captured image;

[0113] S703, finding local extreme points in each difference pyramid, and regarding the local extreme points not less than the set extreme threshold as the key feature points;

[0114] S704, performing sub-pixel localization on the key feature points by using Taylor expansion to determine the coordinates of each key feature point;

[0115] S705, for each key feature point, calculate the gradient histogram in the area around the key feature point, obtain the peak value of the histogram as the main gradient direction, and align the calculation direction of the feature descriptor to the main gradient direction.

[0116] Among them, the step of constructing the Gaussian pyramid of the captured image in S701 includes: performing convolution processing on the captured image by using Gaussian filtering to generate a smoothed image. Downsample the smoothed image to generate a Gaussian pyramid image of the next lower resolution level. Repeat the above steps for each generated Gaussian pyramid image of each level to construct a Gaussian pyramid with a preset number of levels.

[0117] The steps of constructing the Gaussian pyramid of the tile mask map are consistent with the specific process of constructing the pyramid of the captured image.

[0118] The construction formula of the Gaussian pyramid is:

[0119] G(x, y, σ) = I(x, y) * G(x, y, σ0)

[0120] Where G(x, y, σ) represents the image on the σ-th layer of the pyramid, I(x, y) represents the original image, G represents the convolution operation, G(x, y, σ0) represents the Gaussian function, and σ0 represents the initial scale.

[0121] The steps of generating the difference pyramid of the tile mask map according to the tile mask map in S702 include: Upsampling the Gaussian pyramid image of the previous level of the Gaussian pyramid image of the current level of the tile mask map to generate the first Gaussian pyramid image of the tile mask map. Subtracting the first Gaussian pyramid image from the Gaussian pyramid image of the current level to generate the difference image of the corresponding tile mask map. Repeat the above steps until the difference pyramid of the tile mask map with the same number of layers as the Gaussian pyramid is generated.

[0122] The specific process of constructing the difference pyramid of the captured image is consistent with the specific process of constructing the difference pyramid of the tile mask map.

[0123] The construction formula of the difference pyramid is:

[0124] D(x, y, σ) = G(x, y, k·σ) - G(x, y, σ)

[0125] Where D(x, y, σ) represents the image on the σ-th layer of the difference pyramid, G(x, y, k·σ) represents the Gaussian pyramid image with a scale of k·σ, and k represents the scale factor.

[0126] Exemplarily, the tile mask map and the captured image are subjected to multiple Gaussian blur and downsampling operations on the image to obtain the Gaussian pyramid image A of the tile mask map with different resolutions and the Gaussian pyramid image B of the captured image. According to the difference formula and the Gaussian pyramid image A, the difference pyramid A is generated, and according to the difference formula and the Gaussian pyramid image B, the difference pyramid B is generated. Then, local extreme points, i.e., local maximum or minimum values, are found in the difference pyramid. Whether a point is a local extreme point can be determined by comparing the value of each pixel point with the values of its surrounding pixel points. For example, in a 3x3 neighborhood, a pixel point is selected. If the value of this pixel point is larger or smaller than the values of the other 8 pixel points, then this pixel point is a local extreme point. In addition, since the pixel values at the edges change greatly, it is easy to generate misdetected key points. Therefore, edge response points need to be eliminated. Whether a point is an edge response point can be judged by calculating the determinant and trace of the Hessian matrix. If the ratio of the determinant to the trace is greater than a certain threshold, then this point is considered an edge response point and should be excluded.

[0127] Next, the points that meet the preset threshold are selected from the already obtained local extreme points and determined as key feature points. Methods such as Taylor expansion are used to perform sub-pixel localization on the key feature points to determine the exact coordinates of each key feature point, improving the matching accuracy and stability of the key feature points.

[0128] Finally, for each key feature point, the gradient histogram is calculated in the surrounding area to obtain the peak value of the histogram as the main gradient direction. Then, the calculation direction of the feature descriptor is aligned to the main gradient direction to ensure the rotational invariance of the feature descriptor. A feature descriptor with good robustness is generated for subsequent matching and recognition of key feature points.

[0129] The method of this application ensures the coordinate accuracy of the key feature points, improves the accuracy of subsequent formation of matching pairs, and overall improves the accuracy of target positioning.

[0130] In an implementable embodiment of this application, the matching of the key feature points of the tile mask map and the captured image in S602 to obtain multiple key feature point matching pairs includes:

[0131] Based on the feature distances between the key feature points of the tile mask map and the captured image, multiple key feature point matching pairs are determined.

[0132] Due to factors such as noise, occlusion, and repeated structures in the image, there may be some incorrect matches in the process of matching key feature points. The incorrectly matched key feature point matching pairs are screened out through screening.

[0133] Exemplarily, first, calculate the feature distances between each pair of feature points. This is achieved by calculating the Euclidean distance or cosine similarity between feature vectors. Assume that the feature distances between all feature points have been calculated, and now it is necessary to determine which feature points are matching pairs.

[0134] A commonly used method is to use the nearest neighbor matching algorithm, that is, for each feature point, find the feature point in the tile mask graph that is closest to it, and take them as a matching pair. This process can be implemented with the following steps:

[0135] For each feature point in the captured image, calculate the distance between it and all feature points in the tile mask graph.

[0136] For each feature point in the captured image, find the feature point in the tile mask graph that is closest to it, and record the distance between them.

[0137] For each feature point in the captured image, if its nearest neighbor distance is less than a certain threshold, then take it and the corresponding feature point in the tile mask graph as a matching pair.

[0138] For example, assume that 4 key feature points have been extracted from the captured image and the tile mask graph respectively, and their feature descriptors have been calculated. After calculating the feature distances between them, the following matrix is obtained: Next, we use the nearest neighbor matching algorithm to determine the matching pairs. Assume that the threshold we set is 2.0, then according to the above matrix, the following matching pairs can be obtained:

[0139] Key feature point 1 and key feature point C, key feature point 2 and key feature point D, key feature point 3 and key feature point A, key feature point 4 and key feature point B. After obtaining all the key feature point matching pairs, use the RANSAC algorithm to filter the above all matching pairs. Finally, the filtered matching pairs can be used to obtain the mapping relationship between the tile mask graph and the captured image subsequently.

[0140] This application improves the accuracy of obtaining the mapping relationship between the tile mask graph and the captured image through this step.

[0141] As Figure 8 shown, in the implementable embodiments of this application, the tile icon calibration method based on image registration further includes:

[0142] S801, use interpolation mean to fill in the missing values in the captured image, find the nearest points by traversing in a top-left to bottom-right cycle, and perform interpolation averaging on the found left-right pairs or up-down pairs of points;

[0143] S802. If no paired points with three - dimensional position information are found after searching from top - left to bottom - right, assign the nearest non - empty value in the vicinity to this point.

[0144] Exemplarily, traverse each pixel in the captured image. If a missing pixel value is found, it needs to be filled. Search for the nearest point by traversing in a top - left to bottom - right loop. That is, starting from the missing pixel point, search for the nearest non - missing pixel point in the top - left, top - right, bottom - left, and bottom - right directions in sequence. If paired points (i.e., non - missing pixel points are found both horizontally and vertically) are found, the two points can be interpolated and averaged, and the average value is assigned to the missing pixel point. If no paired points with three - dimensional position information are found after searching from top - left to bottom - right, assign the nearest non - empty value in the vicinity to this point. If no paired non - missing pixel points are found during the top - left to bottom - right search, that is, interpolation averaging cannot be performed, the following measures can be taken:

[0145] Assign the nearest non - empty value in the vicinity to this point. That is, search for the nearest non - missing pixel point around the missing pixel point and assign its value to the missing pixel point.

[0146] Specifically, assume there is an image as Figure 9 , in which some pixel values are missing, and it is desired to fill them by the method of interpolation mean. Each pixel in the image can be traversed according to the above steps to find the nearest non - missing pixel point and perform interpolation filling. If no paired non - missing pixel points are found during the top - left to bottom - right search, the nearest non - empty value in the vicinity can be assigned to this point to complete the filling of the missing pixel. The filled image is shown in Figure 10 .

[0147] Specifically, assume that in the captured image of a section of road, there are missing pixel values. The captured photo is filled and improved by the method of difference mean. First, traverse all the pixels in the captured image to find the non - missing pixel point closest to the missing value, calculate the average value of the nearest left - right pair of non - missing pixel points, and take this average value as the value of the missing pixel point. Repeat this process until all the missing pixel points in the captured image are filled.

[0148] This step of the present application can effectively handle the problem of missing pixel values in the image and ensure the integrity of the captured image.

[0149] Figure 11 FIG. 23 is a schematic structural diagram of a device for calibrating tile icons based on image registration provided by an embodiment of the present application. For the sake of clarity, only the parts related to the embodiment of the present application are shown.

[0150] A device 1100 for calibrating tile icons based on image registration may specifically include the following modules:

[0151] An acquisition module 1101 is configured to acquire a high-precision map of a preset road section and a captured image of a roadside camera. The high-precision map includes tile map units of multiple levels, and adjacent higher-level tile map units include corresponding multiple lower-level tile map units.

[0152] A tile mask map generation module 1102 is configured to generate a tile mask map according to the base station longitude and latitude and tile map units of each level.

[0153] A calibration module 1103 is configured to calibrate the external parameters of the roadside camera according to the tile mask map and the captured image.

[0154] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0155] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0156] Figure 12 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 12 shown, the electronic device 3000 of this embodiment includes: at least one processor 3001 ( Figure 12 only one is shown), a memory 3002, and a computer program 3003 stored in the memory 3002 and executable on at least one processor 3001. When the processor 3001 executes the computer program 3003, the steps in the above embodiments are implemented.

[0157] The processor 3001 may be a Central Processing Unit (CPU), and the processor 3001 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0158] In some embodiments, the memory 3002 may be an internal storage unit of the electronic device 3000, such as the hard disk or memory of the electronic device 3000. In other embodiments, the memory 3002 may also be an external storage device of the electronic device 3000, such as a plug-in hard disk equipped on the electronic device 3000, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 3002 may also include both the internal storage unit and the external storage device of the electronic device 3000. The memory 3002 is used to store an operating system, application programs, Boot Loader data, and other programs, such as the program code of a computer program. The memory 3002 may also be used to temporarily store data that has been output or is to be output.

[0159] The embodiments of the present application also provide a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments can be implemented.

[0160] The embodiments of the present application provide a computer program product, and when the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above method embodiments when executed.

[0161] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of this application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0162] In the above embodiments, the descriptions of the various embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0163] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0164] In the embodiments provided in this application, it should be understood that the disclosed device / network device and method can be implemented in other ways. For example, the device / network device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0165] The unit described as a separation component may or may not be physically separated. The component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for calibrating tile icons based on image registration, characterized in that, Including: Obtain the high-precision map of a preset road section and the captured image of the roadside camera. The high-precision map includes tile map units of multiple levels, and adjacent higher-level tile map units include corresponding multiple lower-level tile map units; Generate a tile mask map according to the base station longitude and latitude and each level of tile map unit; Calibrate the external parameters of the roadside camera according to the tile mask map and the captured image.

2. The method according to claim 1, wherein The generation steps of the high-precision map include: Project the surface points of the preset road section into a planar map through Mercator projection; Divide the planar map into tile map units of multiple levels to generate the high-precision map.

3. The method according to claim 1, wherein Generating a tile mask map according to the base station longitude and latitude and each level of tile map unit includes: Obtain the tile map units within a set range according to the base station longitude and latitude and the base station sensing distance; Stitch the filtered tile map units to obtain a high-precision map within a set range; Erase the non-concerned areas in the stitched high-precision map to generate a tile map mask map.

4. The method according to claim 1, wherein The calibrating the external parameters of the roadside camera according to the tile mask map and the captured image includes: Perform image registration on the tile mask map and the captured image to determine the coordinate mapping relationship between the tile mask map and the captured image; Calibrate the external parameters of the roadside camera according to the coordinate mapping relationship between the tile mask map and the captured image.

5. The method according to claim 4, wherein Performing image registration on the tile mask map and the captured image to determine the coordinate mapping relationship between the tile mask map and the captured image includes: Calculate the key feature points of the tile mask map and the captured image; Match the key feature points of the tile mask map and the captured image to obtain multiple key feature point matching pairs; Calculate the coordinate mapping relationship between the tile mask map and the captured image according to all the key feature point matching pairs.

6. The method according to claim 5, wherein The calculating the key feature points of the tile mask map and the captured image includes: Construct the Gaussian pyramids of the tile mask map and the captured image; Generate the difference pyramid of the tile mask map according to the tile mask map, and generate the difference pyramid of the captured image according to the Gaussian pyramid of the captured image; Find the local extreme points in each difference pyramid, and use the local extreme points not less than the set extreme value threshold as the key feature points; Perform sub-pixel localization on the key feature points by using Taylor expansion to determine the coordinates of each key feature point; For each key feature point, calculate the gradient histogram in the area around the key feature point, obtain the peak value of the histogram as the main gradient direction, and align the calculation direction of the feature descriptor to the main gradient direction.

7. The method according to claim 1, characterized in that The matching the key feature points of the tile mask map and the captured image to obtain multiple key feature point matching pairs includes: Determine multiple key feature point matching pairs based on the feature distance between the key feature points of the tile mask map and the captured image.

8. The method according to claim 1, wherein The method further includes: Fill the missing values in the captured image by using interpolation mean, search for the nearest points by traversing in a top-left to bottom-right cycle, and perform interpolation averaging on the found left-right pairs or up-down pairs. If no paired points with three-dimensional position information are found after searching from the upper left to the lower right, the nearest non-empty value in the vicinity is assigned to the point.

9. An apparatus for calibrating tile icons based on image registration, characterized in that, Including: An acquisition module that acquires a high-precision map of a preset road section and a captured image of a roadside camera. The high-precision map includes tile map units of multiple levels, and an adjacent higher-level tile map unit includes corresponding multiple lower-level tile map units; A tile mask map generation module that generates a tile mask map according to the base station's longitude and latitude and tile map units of each level; A calibration module that calibrates the external parameters of the roadside camera according to the tile mask map and the captured image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images