Digital map and infrared image accurate registration method based on contour line matching and application thereof

Abstract

The invention discloses a digital map and infrared image accurate registration method based on contour line matching, which comprises the following steps of: 1, acquiring an infrared image in front of an airplane as an image to be registered, and taking a terrain simulation image of an airborne digital map as a reference image; 2, mapping the infrared image into a terrain simulation image, and carrying out LOD refinement on a mapping area of the infrared image to obtain the terrain simulation image fused with a texture mapping; 3, respectively extracting contour lines L1 and L2 from the infrared image and the terrain simulation image fused with the texture map by using a contour line extraction method; and 4, adjusting data points on the image by using external parameters, and then carrying out precise registration on L1 and L2. According to the method provided by the invention, the detection problem of a given curve in an original image can be converted into the problem of searching a peak value in a parameter space, the accurate registration of a virtual image and a real shot image is realized, and the environment perception capability of a pilot is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion registration processing, and in particular to a method for accurate registration of digital maps and infrared images based on contour matching and its application. Background Art

[0002] An onboard digital map is a two- or three-dimensional map installed on an aircraft, generated using computer technology based on a map information database. It can be displayed on a monitor to assist pilots in navigation and enhance their awareness of the surrounding environment. Its advantages are large coverage, clarity, and unaffected by visibility, such as at night. Its disadvantages are that map processing is constrained by factors such as aircraft navigation positioning errors, attitude errors, and onboard processing delays, resulting in delays or positional deviations. Infrared images, on the other hand, are images of the aircraft's forward direction captured by onboard infrared sensors. While they offer realism, they are limited in coverage, are affected by visibility, and are less effective at night. Therefore, to enhance pilot perception, precise matching of the virtual (digital map) and the real scene (infrared image) is necessary.

[0003] Currently, image fusion algorithms include methods based on multi-scale transformations, sparse representation, neural networks, gradient pyramid decomposition, and wavelet transforms. However, pyramid decomposition is not directional, and the quality of the fused image is generally unsatisfactory. Wavelet transform decomposes the image in the frequency domain and is directional. Fusion of image components in different frequency domains helps improve the visual quality of the fused image. Sparse representation is a very effective image representation theory. Unlike multi-scale fusion methods, sparse representation-based fusion algorithms use image data dictionaries to sparsely represent the source images and achieve image fusion. In recent years, image fusion methods based on deep learning have received increasing attention. However, onboard processing resources are limited, and real-time, dynamic registration is required, so efficient and resource-saving methods are needed.

[0004] Contours and edges are important image features. For example, airports, mountains, and rivers can all be quickly matched by extracting their contours. These methods primarily include edge-based, region-based, and motion-based contour extraction methods, as well as graph-cut, perceptual organization, and active contours-based methods. Registration is then performed based on shape and state space.

[0005] The key characteristics of airborne digital maps and infrared imagery are that foreground data (mountains, buildings, and other terrain) is clearly distinguishable from background data (sky), with similar size and proportion, and distinct, single, prominent contours. Furthermore, due to varying weather conditions and acquisition equipment, the brightness and contrast of images collected at the same location, whether from electro-optical imagery or from an airborne mapping system, can vary. Consequently, existing technologies for key feature point extraction and matching suffer from inaccurate registration. Summary of the Invention

[0006] The purpose of this invention is to provide a precise registration method for digital maps and infrared images based on contour matching. The core concept is: when pre-loaded data includes high-resolution digital orthophotos (DOM), the high-resolution DOM data is used to register with airborne infrared imagery, thereby indirectly obtaining a corrected registration relationship between the infrared image and the topographic map. Furthermore, a matching method based on edge features is used to achieve rapid registration and fusion of the DOM data and the infrared imagery.

[0007] To achieve the above object, the present invention provides a method for accurately registering a digital map and an infrared image based on contour matching, which is characterized by comprising the following steps:

[0008] Step 1: Obtain an infrared image in front of the aircraft as the image to be registered, and use the terrain simulation image of the onboard digital map as the reference image;

[0009] Step 2: Map the infrared image to the terrain simulation image, and perform LOD refinement on the infrared image mapping area to obtain the terrain simulation image fused with the texture map;

[0010] Step 3: Use the contour line extraction method to extract contour lines L1 and L2 from the infrared image and the terrain simulation image fused with texture mapping respectively;

[0011] Step 4: Adjust the data points on the image using external parameters and then accurately align L1 and L2.

[0012] Furthermore, the contour line extraction method described in step 3 includes the following steps:

[0013] Step 41: For the terrain simulation image C2, a grayscale image is constructed using the B component in the RGB image. For the infrared image C1, its grayscale image is directly obtained.

[0014] Step 42: Segment the obtained grayscale image using the large law, setting the foreground pixels to 1 and the background pixels to 0;

[0015] Step 43: traverse from the 0th column to the last column of the segmented grayscale image, and at the same time traverse from the 0th row to the last row for the data in each column;

[0016] Step 44: In each column, calculate the intensity of each pixel and subtract the current intensity from the intensity of the previous pixel and the next pixel to obtain the gradient.

[0017] Step 45: When the gradient exceeds the preset threshold, the pixel is defined as a candidate point in this column;

[0018] Step 46: Repeat steps 43-45 until all columns are traversed, and all the candidate points extracted are used as contour points to obtain the terrain contours L1 and L2 of C1 and C2 respectively, and each contour point n in L1 and L2 i All are two-dimensional data (u i , v i ), where u i , v i Represents the pixel coordinates of the image.

[0019] Furthermore, the specific steps of step 5 include:

[0020] Step 51: Take the origin of the image coordinate system as the zero point;

[0021] Step 52: Each contour point (u i , v i ) are transformed into:

[0022]

[0023] Among them, ρ i is point n i Distance to the origin, θ i is the angle;

[0024] Step 53: Get the point set of the contour line and The rigid body transformation is:

[0025] L1=RL2+T

[0026] Among them, R is the rotation transformation and T is the translation transformation;

[0027] Step 54: Construct the objective optimization function that minimizes the L1 and L2 average errors:

[0028]

[0029] Step 55: Solve the above equation to obtain the optimal transformation relationships R and T between L1 and L2, and then use the ICP method to perform alignment.

[0030] The digital map and infrared image precise registration system based on contour matching is implemented based on the digital map and infrared image precise registration method based on contour matching, and is characterized by comprising:

[0031] The image acquisition module is used to acquire the infrared image in front of the aircraft, using the terrain simulation image of the onboard digital map as the reference image and the infrared image as the image to be registered;

[0032] Image mapping module, used to map infrared images into terrain simulation images;

[0033] An image enhancement module is used to subdivide the grid of the infrared image mapping area of ​​the terrain simulation image to obtain a terrain simulation image fused with texture mapping;

[0034] A contour line extraction module is used to extract contour lines from infrared images and terrain simulation images fused with texture maps;

[0035] The contour registration module is used to perform registration based on the extracted contour lines.

[0036] An electronic device, characterized in that it includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for precise registration of digital maps and infrared images based on contour matching as described in any one of claims 1 to 3.

[0037] A computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the method for accurate registration of a digital map and an infrared image based on contour matching as described in any one of claims 1 to 3 is implemented.

[0038] A computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the method for accurate registration of a digital map and an infrared image based on contour matching as described in any one of claims 1 to 3 is implemented.

[0039] Therefore, the present invention adopts the above-mentioned digital map and infrared image precise registration method based on contour matching, which has the following beneficial effects:

[0040] Aiming at the characteristic that infrared sensors detect in the front and lower direction of the aircraft, the present invention proposes a method for extracting contour lines and precise matching. By designing a structural similarity measurement discriminant function and matching the contour information based on line feature association technology, the two-dimensional rotation and translation error between the image and the terrain data is obtained, which is used to correct the image deviation. The problem of detecting a given curve in the original image is thus transformed into the problem of finding a peak in the parameter space, thus achieving precise alignment between the virtual image and the real image, and improving the pilot's environmental perception ability.

[0041] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The flowchart of the method proposed in the present invention is shown in FIG.

[0043] Figure 2 The original data of the infrared image.

[0044] Figure 3 The raw data for rendering the image.

[0045] Figure 4 It is the process of vertical gradient contour extraction.

[0046] Figure 5 The result of contour extraction.

[0047] Figure 6 The matching results between digital map and infrared image. DETAILED DESCRIPTION

[0048] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art will make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the application.

[0049] The present invention proposes a method for accurate registration of digital maps and infrared images based on contour matching, the specific steps of which include:

[0050] 1. Registering images and reference images

[0051] When performing registration, the terrain simulation image of the airborne digital map is used as the reference image, and the infrared image is used as the image to be registered.

[0052] Because both observed infrared images and terrain data simulation images have weak texture information (terrain data simulation images lack texture), corresponding points cannot be found through feature point extraction and matching. Furthermore, due to errors in the onboard internal and external parameters, they are insufficient to support high-precision data registration. Furthermore, because the grid resolution of terrain models is limited to tens of meters, while the spatial resolution of infrared images can reach the centimeter level, this resolution difference creates a significant spatial scale imbalance.

[0053] 2. Refinement of texture mapping area for local LOD

[0054] Due to the significant difference in resolution between terrain images and infrared images, a mosaic region LOD rendering scheme was used to refine the terrain model areas mapped by the infrared image. Different areas of the terrain were described and rendered with varying levels of detail, depending on the complexity of the terrain and the characteristics of the human eye's observation of the terrain.

[0055] In the infrared image mapping area, LOD technology is used to draw the terrain, increasing the number of triangles in the mapping area and enriching the depth representation of the texture mapping. The more triangles there are, the higher the accuracy of the terrain description. For other areas, the number of triangles drawn is kept at a low level, and the terrain description accuracy is on par with the accuracy of the original terrain map. The mesh subdivision of the infrared image mapping area uses the C proposed in the prior art. 2 Continuous triangular mesh subdivision method, which can obtain C at the singular point 1 Continuous, and C in other positions 2 Continuous mesh surface. The subdivision surface is obtained by inserting a point in each triangle, connecting the point with the vertex of the triangle, and then performing an edge flip operation. After subdivision, a semi-regular mesh is generated, with a new vertex degree of 6, while the original vertex degrees remain unchanged. After two subdivisions, each initial triangle is split into nine new triangles, performing a ternary split operation. After fusing the texture maps, pilots can see a terrain model with infrared image information, greatly improving situational awareness in low-visibility environments and providing diverse information from a single window.

[0056] 3. Contour extraction

[0057] By extracting the contours of mountains and rivers from the infrared image and those from the digital map, the relative position of the two can be determined for registration. Since the pilot's perspective is downward and forward in flight, there is a horizon line between the sky and the earth, which can be used as the contour line for matching.

[0058] When a pilot looks down from the air, the small foreground area is an infrared image of mountains, buildings, and other objects captured by the infrared sensor, while the background is a virtual scene generated from large-scale geographic data. The image contains two types of pixels (foreground pixels and background pixels). Therefore, this invention uses inter-class variance (the Great Law) to segment the background and foreground.

[0059] After obtaining the infrared image C1 and the digital map image C2, in order to solve the problem of limited onboard computing resources, the present invention uses the vertical gradient method to extract the contour line, traverse the image with the least resources, and construct the contour line. First, the grayscale images of C1 and C2 must be obtained, such as Figure 2 As shown in , for C1, the duality of its data is more obvious, and its grayscale image can be directly obtained for single-channel images such as infrared images; Figure 3 As shown, for the image C2 from the digital map service of the aircraft, a colored three-dimensional rendering image is usually obtained. Since the color and texture can be defined by oneself and are not affected by weather, etc., a grayscale image is constructed using the blue component (B component) in the RGB image.

[0060] like Figure 4 As shown, the extraction of contour lines includes the following steps:

[0061] 1) Obtain a grayscale image;

[0062] 2) Segment the grayscale image using the big law, and set the foreground pixel value to 1 and the background pixel value to 0;

[0063] 3) Traverse the segmented grayscale image from the 0th column to the last column;

[0064] 4) For each column of data, traverse from row 0 to the last row;

[0065] 5) In each column, calculate each pixel a i The intensity of a i With the previous pixel a i-1 and a i With the next pixel a i+1 The difference in intensity and the gradient are obtained;

[0066] 6) If the gradient exceeds the preset threshold, the pixel is defined as a candidate point in this column;

[0067] 7) Traverse all columns, obtain candidate points for each column, and use the candidate point set as the contour point set.

[0068] Finally, the contour point sets L1 and L2 corresponding to the two images C1 and C2 are obtained through the above steps, and each contour point n in L1 and L2 i All are two-dimensional data (u i, v i ), where u i , v i Represents the pixel coordinates of the image. The contour line results extracted from the infrared image are as follows Figure 5 shown.

[0069] 4. Accurate registration of image contours

[0070] Since the field of view of the rendering software is different from that of the aircraft, the image pixels may be different. By calculating the translation and rotation errors of the terrain contours L1 and L2 on the image plane and normalizing them to the same coordinate system, the aircraft can be accurately positioned.

[0071] Since the data for both images C1 and C2 are originally in a pixel coordinate system, the coordinates of the aforementioned contour points are also pixel locations. The present invention maps the infrared image data onto a suitable terrain rendering through coordinate transformation. The terrain rendering's coordinate system is coaxial with the camera's coordinate system, but with a larger field of view. Based on the FOV of the camera models corresponding to the two images, the infrared image and the terrain rendering can be superimposed through coordinate transformation and scaling.

[0072] Finally, after the contour is obtained and the data points on the image are adjusted using external parameters, the iterative closest point (ICP) method can be used for registration.

[0073] The origin of the image coordinate system is the zero point (i.e. the point where u=0, v=0). Then every point (u i , v i ) can be transformed into:

[0074]

[0075] Among them, ρ i is point n i Distance to the origin, θ i is the angle.

[0076] Then the point set Hedianji The rigid body transformation is as follows:

[0077] L1=RL2+T

[0078] The matching goal is to find the point set {a i}∈L1 and {b i The transformation R (rotation) and T (translation) that minimizes the Euclidean distance of}∈L2, that is:

[0079]

[0080] By solving the above equation, we can get the optimal transformation relationship R and T between the two images, which means that the rotation and translation of the set of contour lines of L1 can finally achieve the lowest average error with the set of contour lines of L2.

[0081] According to the registration correspondence between the two, the accuracy of the carrier platform and equipment internal parameters used in generating terrain simulation images is corrected.

[0082] Example

[0083] In order to verify the effect of the method proposed in the present invention, in the algorithm simulation, the photoelectric image data with a smaller intermediate field of view is first translated upward by 20 pixels to generate an error in the aircraft elevation. Then, the error can be eliminated by matching, resulting in a result that the photoelectric image is consistent with the rendered terrain image. The fusion effect is as follows: Figure 6 shown.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for accurate registration of digital maps and infrared images based on contour matching, characterized in that: The following steps are involved: Step 1: Obtain an infrared image in front of the aircraft as the image to be registered, and use the terrain simulation image of the onboard digital map as the reference image; Step 2: Map the infrared image to the terrain simulation image, and perform LOD refinement on the infrared image mapping area to obtain the terrain simulation image fused with the texture map; Step 3: Use the contour line extraction method to extract contour lines L1 and L2 from the infrared image and the terrain simulation image fused with texture mapping respectively; Step 4: Adjust the data points on the image using external parameters and then accurately align L1 and L2.

2. The method for accurate registration of digital maps and infrared images based on contour matching according to claim 1 is characterized in that: The contour line extraction method described in step 3 includes the following steps: Step 41: For the terrain simulation image C2, a grayscale image is constructed using the B component in the RGB image. For the infrared image C1, its grayscale image is directly obtained. Step 42: Segment the obtained grayscale image using the large law, setting the foreground pixels to 1 and the background pixels to 0; Step 43: traverse from the 0th column to the last column of the segmented grayscale image, and at the same time traverse from the 0th row to the last row for the data in each column; Step 44: In each column, calculate the intensity of each pixel and subtract the current intensity from the intensity of the previous pixel and the next pixel to obtain the gradient. Step 45: When the gradient exceeds the preset threshold, the pixel is defined as a candidate point in this column; Step 46: Repeat steps 43-45 until all columns are traversed, and all the candidate points extracted are used as contour points to obtain the terrain contours L1 and L2 of C1 and C2 respectively, and each contour point n in L1 and L2 i All are two-dimensional data (u i , v i ), where u i , v i Represents the pixel coordinates of the image.

3. The method for accurate registration of digital maps and infrared images based on contour matching according to claim 2, characterized in that: The specific steps of step 5 include: Step 51: Take the origin of the image coordinate system as the zero point; Step 52: Each contour point (u i , v i ) are transformed into: Among them, ρ i is point n i Distance to the origin, θ i is the angle; Step 53: Get the point set of the contour line and The rigid body transformation is: L1=RL2+T Among them, R is the rotation transformation and T is the translation transformation; Step 54: Construct the objective optimization function that minimizes the L1 and L2 average errors: Step 55: Solve the above equation to obtain the optimal transformation relationships R and T between L1 and L2, and then use the ICP method to perform alignment.

4. A system for accurately registering digital maps and infrared images based on contour matching, implemented based on the method for accurately registering digital maps and infrared images based on contour matching according to claim 1, characterized in that: include: The image acquisition module is used to acquire the infrared image in front of the aircraft, using the terrain simulation image of the onboard digital map as the reference image and the infrared image as the image to be registered; Image mapping module, used to map infrared images into terrain simulation images; An image enhancement module is used to subdivide the grid of the infrared image mapping area of ​​the terrain simulation image to obtain a terrain simulation image fused with texture mapping; A contour line extraction module is used to extract contour lines from infrared images and terrain simulation images fused with texture maps; The contour registration module is used to perform registration based on the extracted contour lines.

5. An electronic device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for precise registration of digital maps and infrared images based on contour matching as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for accurate registration of a digital map and an infrared image based on contour matching according to any one of claims 1 to 3 is implemented.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for accurate registration of digital maps and infrared images based on contour matching as described in any one of claims 1 to 3 is implemented.

Images