Coarse-fine two-stage registration method and device based on visible light image and infrared image

Abstract

The invention discloses a coarse-fine two-stage registration method based on a visible light image and an infrared image, and belongs to the technical field of image processing. The method comprises the following steps: establishing a first 3DGS model based on visible light data and generating visible light point cloud data, and establishing a second 3DGS model based on infrared data and generating infrared point cloud data; feature values of the visible light point cloud data and the infrared point cloud data are extracted by adopting a KPConv network and rough matching is carried out to obtain rough matching parameters, and the rough matching parameters comprise a first scaling factor, a first rotation matrix and a first translation vector; a visible light image and an infrared image are generated based on the rough matching parameters, all angular points in the visible light image and the infrared image are extracted through the FAST algorithm, a visible light angular point set and an infrared angular point set are obtained, fine matching is conducted, fine matching parameters are obtained, and the fine matching parameters comprise a first scaling factor, a second rotation matrix and a second translation vector; the target image is obtained based on the fine matching parameters, and the matching degree of the visible light 3DGS model and the infrared 3DGS model is improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a coarse-fine two-stage registration method and device based on visible light images and infrared images. Background Art

[0002] The rapid development of 3DGS (3D Geometric Surface) technology has demonstrated significant advantages in 3D modeling and scene reconstruction in fields such as aerial surveying, autonomous driving, and augmented reality. It is particularly capable of generating high-fidelity, detailed 3D models when dealing with complex lighting, transparent materials, and dynamic scenes. However, in aerial survey scenarios where multiple drone sensors (visible light and infrared cameras) collaborate, achieving high-precision cross-modal data registration and fusion remains a core challenge hindering the further application of 3DGS technology.

[0003] Existing methods for registering infrared and visible light data primarily utilize local alignment techniques based on feature matching and global optimization methods based on mutual information. The former extracts local features such as edges and corners from cross-modal images for matching. However, in scenarios with significant differences in spectral response, the discriminability of feature descriptors is insufficient, leading to mismatches and registration failures. The latter achieves alignment by maximizing the statistical dependence between the two images. While robust to global grayscale distribution, it is prone to registration errors due to local non-rigid deformations when dealing with dynamic targets or complex geometric structures in large-scale aerial survey scenes.

[0004] Furthermore, existing methods are mostly limited to two-dimensional image registration and fail to consider geometric consistency constraints in three-dimensional space. This results in spatial topological misalignment between the reconstructed visible light and infrared 3DGS models, making it difficult to directly fuse them into a unified cross-modal 3D representation. Existing methods also suffer from low matching accuracy, efficiency, and speed when matching visible light and infrared 3DGS models. Summary of the Invention

[0005] This application aims to solve at least one of the technical problems existing in the prior art. To this end, this application proposes a coarse-fine two-stage registration method and device based on visible light images and infrared images, which improves the fusion of 3DGS models established using multiple sensors.

[0006] In a first aspect, the present application provides a coarse-fine two-stage registration method based on visible light images and infrared images, the method comprising:

[0007] Obtaining visible light data and infrared data, wherein the visible light data is obtained by collecting the target scene using a visible light camera sensor, and the infrared data is obtained by collecting the target scene using an infrared camera sensor;

[0008] Establishing a first 3DGS model based on the visible light data and generating visible light point cloud data, establishing a second 3DGS model based on the infrared data and generating infrared point cloud data;

[0009] Using a KPConv network to extract eigenvalues ​​of the visible light point cloud data and the infrared point cloud data and perform coarse matching to obtain coarse matching parameters, wherein the coarse matching parameters include a first scaling factor, a first rotation matrix, and a first translation vector;

[0010] generating a visible light image and an infrared image based on the coarse matching parameters, and extracting all corner points in the visible light image and the infrared image using the FAST algorithm to obtain a visible light corner point set and an infrared corner point set;

[0011] Performing precise matching on the visible light corner point set and the infrared corner point set to obtain precise matching parameters, wherein the precise matching parameters include a first scaling factor, a second rotation matrix, and a second translation vector;

[0012] Based on the precise matching parameters, the target image is obtained, where the target image is a fusion image of the optical image and the infrared image.

[0013] According to one embodiment of the present application, the extracting feature values ​​of the visible light point cloud data and the infrared point cloud data using the KPConv network and performing rough matching to obtain rough matching parameters includes:

[0014] Performing multi-level uniform downsampling on the visible light point cloud data and the infrared point cloud data based on a KPConv network to obtain a first sampling set and a second sampling set;

[0015] Performing feature extraction on the first sampling set and the second sampling set to obtain a first feature set and a second feature set;

[0016] Performing layer-by-layer feature encoding on the first feature set and the second feature set based on KPConv kernel point convolution and feature pyramid network to obtain a visible light point cloud feature point set and an infrared point cloud feature point set;

[0017] Rough matching is performed based on the visible light point cloud feature point set and the infrared point cloud feature point set to obtain rough matching parameters.

[0018] According to one embodiment of the present application, performing coarse matching based on the visible light point cloud feature point set and the infrared point cloud feature point set to obtain coarse matching parameters includes:

[0019] An objective function is established based on the visible light point cloud feature point set and the infrared point cloud feature point set. The expression of the objective function is as follows:

[0020]

[0021] Where E is the sum of the squared Euclidean distances between all matching point pairs in the visible light point cloud feature point set and the infrared point cloud feature point set, s is the scaling factor, R is the rotation matrix, t is the translation vector, and q is the translation vector. i is the i-th feature point in the visible light point cloud feature point set, p i is the i-th feature point in the infrared point cloud feature point set, N p is the number of feature points in the visible light point cloud feature point set;

[0022] Solving the minimum solution of the objective function to obtain a transformation matrix;

[0023] The transformation matrix is ​​used as a coarse matching parameter.

[0024] According to one embodiment of the present application, extracting all corner points in the visible light image and the infrared image using the FAST algorithm to obtain a visible light corner point set and an infrared corner point set includes:

[0025] Traverse all pixels in the visible light image, and use the FAST algorithm to determine whether each pixel is a corner point. All pixels that are corner points in the visible light image are regarded as the visible light corner point set.

[0026] All pixels in the infrared image are traversed, and for each pixel, the FAST algorithm is used to determine whether the pixel is a corner point. All pixels that are corner points in the infrared image are regarded as the infrared corner point set.

[0027] According to one embodiment of the present application, the using of the FAST algorithm to determine whether the pixel point is a corner point includes:

[0028] Draw a circle with the pixel as the center and the preset radius, and calculate the grayscale values ​​of all the pixels that the circle passes through;

[0029] Determine in sequence whether the difference between the grayscale value of each pixel and the pixel is greater than or equal to a preset threshold, and obtain the target number of pixels greater than or equal to the preset threshold;

[0030] Determine whether the number of targets is greater than half of the total number of pixels. If the number of targets is greater than half of the total number of pixels, the pixel is regarded as a corner point.

[0031] According to one embodiment of the present application, performing precise matching on the visible light corner point set and the infrared corner point set to obtain precise matching parameters includes:

[0032] Calculating feature similarities between the visible light corner point set and the infrared corner point set to obtain initial paired feature points;

[0033] Calculating a basic matrix based on the initial paired feature points;

[0034] The basic matrix is ​​decomposed to obtain precise matching parameters.

[0035] According to one embodiment of the present application, decomposing the basic matrix to obtain precise matching parameters includes:

[0036] a. Select the minimum feature points required by the basic matrix to solve the basic matrix;

[0037] b. Substitute all initial pairing feature points into the obtained basic matrix, determine whether each initial pairing feature point is within the preset error range, and use the initial pairing feature points within the preset error range as inliers to obtain the number of inliers; c. Repeat steps a-b until the number of inliers is greater than the preset number of inliers or the number of iterations reaches the preset number of iterations; d. Decompose the basic matrix with the largest number of inliers to obtain the precise matching parameters.

[0038] In a second aspect, the present application provides a coarse-fine two-stage registration device based on visible light images and infrared images, the device comprising:

[0039] An acquisition module, configured to acquire visible light data and infrared data, wherein the visible light data is acquired by acquiring a target scene through a visible light camera sensor, and the infrared data is acquired by acquiring a target scene through an infrared camera sensor;

[0040] A first processing module is configured to establish a first 3DGS model based on the visible light data and generate visible light point cloud data, and to establish a second 3DGS model based on the infrared data and generate infrared point cloud data;

[0041] A coarse matching module is used to extract eigenvalues ​​of the visible light point cloud data and the infrared point cloud data using a KPConv network and perform coarse matching to obtain coarse matching parameters, wherein the coarse matching parameters include a scaling factor, a rotation matrix, and a translation vector;

[0042] A second processing module is configured to generate a visible light image and an infrared image based on the coarse matching parameters, and extract all corner points in the visible light image and the infrared image using a FAST algorithm to obtain a visible light corner point set and an infrared corner point set;

[0043] A fine matching module, configured to perform fine matching on the visible light corner point set and the infrared corner point set to obtain fine matching parameters, wherein the fine matching parameters include a scaling factor, a rotation matrix, and a translation vector;

[0044] A generating module is used to obtain the target image based on the precise matching parameters, where the target image is a fusion image of the optical image and the infrared image.

[0045] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the coarse-and-fine two-stage registration method based on visible light images and infrared images as described in the first aspect above is implemented.

[0046] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the coarse-fine two-stage registration method based on visible light images and infrared images as described in the first aspect above.

[0047] In the fifth aspect, the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the coarse-fine two-stage registration method based on visible light images and infrared images as described in the first aspect.

[0048] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the coarse-fine two-stage registration method based on visible light images and infrared images as described in the first aspect above.

[0049] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application.

[0050] The present invention provides a coarse-fine two-stage registration method based on visible light images and infrared images, which has the following advantages over the prior art:

[0051] (1) The present invention obtains visible light data and infrared data for 3DGS modeling, and generates visible light point cloud data and infrared point cloud data respectively. The KPConv network is used to perform coarse matching on the point cloud data to obtain coarse matching parameters and generate visible light images and infrared images. The FAST algorithm is used to extract corner point sets in the visible light image and infrared image, and fine matching is performed on them to obtain fine matching parameters. The coarse-fine two-stage registration of infrared and visible light images is realized, which improves the efficiency and accuracy of matching between infrared and visible light data and reduces the registration error.

[0052] (2) The present invention performs multi-level uniform downsampling on visible light point cloud data and infrared point cloud data based on the KPConv network to obtain a first sampling set and a second sampling set, and then performs feature extraction to obtain a first feature set and a second feature set, which can extract more accurate and efficient feature information. The feature set is encoded layer by layer by combining KPConv kernel point convolution and feature pyramid network, which further improves the feature expression ability. Through the coarse matching process, the time and complexity of subsequent matching can be reduced, and the efficiency and accuracy of matching of infrared data and visible light data are improved.

[0053] (3) The present invention uses the FAST algorithm to extract all corner points in visible light images and infrared images, which can effectively obtain feature information in the image, perform corner point detection on visible light images and infrared images respectively, and obtain visible light corner point sets and infrared corner point sets. It can extract more accurate features from visible light images and infrared images, improve the accuracy of corner point detection, achieve faster and more accurate feature alignment, and improve the efficiency and accuracy of matching infrared data and visible light data. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0055] Figure 1 This is one of the flow charts of the coarse-fine two-stage registration method based on visible light images and infrared images provided in an embodiment of the present application;

[0056] Figure 2 This is a schematic diagram of the rough matching process provided in the embodiment of the present application;

[0057] Figure 3 This is a schematic diagram of the precise matching process provided in the embodiment of the present application;

[0058] Figure 4 This is the second flow chart of the coarse-fine two-stage registration method based on visible light images and infrared images provided in an embodiment of the present application;

[0059] Figure 5 This is a matching fusion effect diagram provided in an embodiment of the present application;

[0060] Figure 6 Schematic diagram of the structure of a coarse-fine two-stage registration device based on visible light images and infrared images provided in an embodiment of the present application;

[0061] Figure 7 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0063] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0064] Below, in combination with the accompanying drawings, the coarse-fine two-stage registration method based on visible light images and infrared images, the coarse-fine two-stage registration device based on visible light images and infrared images, the electronic device and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0065] The coarse-fine two-stage registration method based on visible light images and infrared images can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0066] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).

[0067] In the following embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse, and a joystick.

[0068] The embodiment of the present application provides a coarse-fine two-stage registration method based on visible light images and infrared images. The execution subject of the coarse-fine two-stage registration method based on visible light images and infrared images can be an electronic device or a functional module or functional entity in the electronic device that can implement the coarse-fine two-stage registration method based on visible light images and infrared images. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablets, computers, cameras and wearable devices. The coarse-fine two-stage registration method based on visible light images and infrared images provided in the embodiment of the present application is explained below using electronic devices as the execution subject as an example.

[0069] Figure 1 This is one of the flow charts of the coarse-fine two-stage registration method based on visible light images and infrared images provided in the embodiment of the present application, such as Figure 1 As shown, the coarse-fine two-stage registration method based on visible light images and infrared images includes: step 110, step 120, step 130, step 140, step 150 and step 160.

[0070] Step 110: Obtain visible light data and infrared data, wherein the visible light data is obtained by collecting the target scene through a visible light camera sensor, and the infrared data is obtained by collecting the target scene through an infrared camera sensor;

[0071] It is easy to understand that the visible light camera sensor and the infrared camera sensor respectively collect video or image data of the target scene to obtain visible light data and infrared data. The visible light data is the image or video data in the visible light band, and the infrared data is the image or video data in the infrared band.

[0072] Step 120: Establish a first 3DGS model based on the visible light data and generate visible light point cloud data; establish a second 3DGS model based on the infrared data and generate infrared point cloud data;

[0073] It should be noted that the visible light data is reconstructed in three dimensions to generate a first 3DGS (3D GeometricSurface) model, and the three-dimensional coordinates are calculated using stereo matching and triangulation techniques to generate visible light point cloud data. The infrared data is reconstructed in three dimensions to generate a second 3DGS model, and the temperature information in the infrared data is mapped into three-dimensional space to generate infrared point cloud data.

[0074] Step 130: Use a KPConv network to extract eigenvalues ​​of the visible light point cloud data and the infrared point cloud data and perform coarse matching to obtain coarse matching parameters, where the coarse matching parameters include a first scaling factor, a first rotation matrix, and a first translation vector.

[0075] It is easy to understand that KPConv (Kernel Point Convolution) is used to extract features of visible light point cloud data and infrared point cloud data, obtain the characteristic values ​​of visible light point cloud data and the characteristic values ​​of infrared point cloud data, and perform coarse matching to obtain coarse matching parameters, which include a first scaling factor, a first rotation matrix, and a first translation vector.

[0076] The first scaling factor represents the size difference between the visible light point cloud data and the infrared point cloud data, that is, how to scale one of the point cloud data to match the other point cloud data. The first rotation matrix represents the rotation relationship between the visible light point cloud data and the infrared point cloud data, that is, how to rotate one of the point cloud data so that it can align with the other point cloud data. The first translation vector represents the translation relationship between the visible light point cloud data and the infrared point cloud data.

[0077] Step 140: Generate a visible light image and an infrared image based on the coarse matching parameters, and use the FAST algorithm to extract all corner points in the visible light image and the infrared image to obtain a visible light corner point set and an infrared corner point set;

[0078] Furthermore, an image generation method is used to convert the visible light data and the infrared data into corresponding image formats according to the coarse matching parameters to obtain a visible light image and an infrared image.

[0079] The FAST (Features from Accelerated Segment Test) algorithm is an algorithm used to detect corner points (or key points) in an image. Corner points are points with significant features in an image. These points usually have large changes in local areas of the image (such as edge intersections and locations with obvious texture changes). The FAST algorithm is used to extract all corner points in visible light images and infrared images to obtain visible light corner point sets and infrared corner point sets.

[0080] Step 150: Perform precise matching on the visible light corner point set and the infrared corner point set to obtain precise matching parameters, where the precise matching parameters include a first scaling factor, a second rotation matrix, and a second translation vector.

[0081] Finally, in order to further improve the matching of visible light data and infrared data, precise matching is performed based on the visible light corner point set and the infrared corner point set to obtain precise matching parameters. The precise matching parameters include the first scaling factor, the second rotation matrix, and the second translation vector.

[0082] It is worth noting that the first scaling factor in the fine matching parameters is the same as the first scaling factor in the coarse matching parameters.

[0083] Step 160: Based on the precise matching parameters, obtain the target image, which is a fusion image of the optical image and the infrared image.

[0084] According to the coarse-fine two-stage registration method based on visible light images and infrared images provided in the embodiments of the present application, 3DGS modeling is performed by acquiring visible light data and infrared data, and visible light point cloud data and infrared point cloud data are generated respectively. The KPConv network is used to perform coarse matching on the point cloud data to obtain coarse matching parameters and generate visible light images and infrared images. The FAST algorithm is used to extract corner point sets in the visible light image and infrared image, and fine matching is performed on them to obtain fine matching parameters. This realizes coarse-fine two-stage registration of infrared images and visible light images, improves the efficiency and accuracy of matching of infrared data and visible light data, and reduces registration errors.

[0085] In some embodiments, the extracting feature values ​​of the visible light point cloud data and the infrared point cloud data using the KPConv network and performing rough matching to obtain rough matching parameters includes:

[0086] Performing multi-level uniform downsampling on the visible light point cloud data and the infrared point cloud data based on a KPConv network to obtain a first sampling set and a second sampling set;

[0087] Performing feature extraction on the first sampling set and the second sampling set to obtain a first feature set and a second feature set;

[0088] Performing layer-by-layer feature encoding on the first feature set and the second feature set based on KPConv kernel point convolution and feature pyramid network to obtain a visible light point cloud feature point set and an infrared point cloud feature point set;

[0089] Rough matching is performed based on the visible light point cloud feature point set and the infrared point cloud feature point set to obtain rough matching parameters.

[0090] Figure 2 This is a schematic diagram of the rough matching process provided in the embodiment of the present application. Figure 2 As shown in the figure, we first use the KPConv network to perform multi-level downsampling to extract the initial features of each point. After obtaining the point cloud features at different scales, we perform feature matching. Based on the matched point cloud pairs, we calculate the coarse matching parameters of the visible light point cloud data and the infrared point cloud data. The specific process is as follows:

[0091] (1) Point cloud downsampling and feature encoding: Based on the KPConv network, multi-level uniform downsampling is implemented. For the input visible light point cloud data and infrared point cloud data, a point X is selected as the center of the sphere, a sphere is determined with r as the radius, and several core points x are selected within the sphere. Each core point has a weight matrix. For any point within the sphere, the kernel function is used to calculate the weight matrix of the point, and the matrix is ​​used to transform the features of the point. For each point within the sphere, the above method is used to calculate new features, and finally these features are accumulated as the features of point X. The formula is:

[0092]

[0093] Among them, g(x i -x) is the kernel function, f i is the feature after convolution, F * (g(x)) is the characteristic of point x.

[0094] The calculation formula of the kernel function is as follows:

[0095]

[0096] in, is the position of the kth core point, W k is the first weight coefficient, is the second weight coefficient, and the calculation formula of the second weight coefficient is as follows:

[0097]

[0098] (2) KPConv kernel point convolution and feature pyramid network are used within each point cloud scale to realize the layer-by-layer feature encoding of point clouds, and point cloud features at different sampling scales are obtained. Finally, the visible light point cloud feature points after N (for example, 5) times of downsampling are obtained. and infrared point cloud feature points

[0099] In this embodiment, the visible light point cloud data and the infrared point cloud data are uniformly down-sampled at multiple levels based on the KPConv network to obtain a first sampling set and a second sampling set, and then feature extraction is performed to obtain a first feature set and a second feature set, which can extract more accurate and efficient feature information. The feature set is encoded layer by layer by combining KPConv kernel point convolution and feature pyramid network, which further improves the feature expression ability. Through the coarse matching process, the time and complexity of subsequent matching can be reduced, and the efficiency and accuracy of matching of infrared data and visible light data are improved.

[0100] In some embodiments, performing coarse matching based on the visible light point cloud feature point set and the infrared point cloud feature point set to obtain coarse matching parameters includes:

[0101] An objective function is established based on the visible light point cloud feature point set and the infrared point cloud feature point set. The expression of the objective function is as follows:

[0102]

[0103] Where E is the sum of the squared Euclidean distances between all matching point pairs in the visible light point cloud feature point set and the infrared point cloud feature point set, s is the scaling factor, R is the rotation matrix, t is the translation vector, and q is the translation vector. i is the i-th feature point in the visible light point cloud feature point set, p i is the i-th feature point in the infrared point cloud feature point set, N p is the number of feature points in the visible light point cloud feature point set;

[0104] Solving the minimum solution of the objective function to obtain a transformation matrix;

[0105] The transformation matrix is ​​used as a coarse matching parameter.

[0106] It is easy to understand that after obtaining the relevant matching feature points of the visible light point cloud and the infrared point cloud at different scales, the transformation relationship {s, R, t} between the two point clouds is solved, where s is the scaling factor, R is the rotation matrix, and t is the translation vector. This problem can be converted into calculating the feature point set P = {p1, p2, ..., p n} and Q={q1,q2,…,q n}, the solution {s, R, t} that minimizes the objective function, the objective function is the sum of the squared Euclidean distances between all matching point pairs in the visible light point cloud feature point set and the infrared point cloud feature point set, and then the calculated transformation matrix is ​​used as the coarse matching parameter.

[0107] In this embodiment, by establishing an objective function, the geometric relationship between visible light and infrared point cloud data is quantified, and the minimum solution of the objective function is solved to obtain a transformation matrix, which can better describe the transformation relationship between visible light point cloud data and infrared point cloud data, achieve a rough matching of infrared data and visible light data, reduce errors caused by differences between different sensors and data noise, and improve the robustness and efficiency of the alignment.

[0108] In some embodiments, extracting all corner points from the visible light image and the infrared image using the FAST algorithm to obtain a visible light corner point set and an infrared corner point set includes:

[0109] Traverse all pixels in the visible light image, and use the FAST algorithm to determine whether each pixel is a corner point. All pixels that are corner points in the visible light image are regarded as the visible light corner point set.

[0110] All pixels in the infrared image are traversed, and for each pixel, the FAST algorithm is used to determine whether the pixel is a corner point. All pixels that are corner points in the infrared image are regarded as the infrared corner point set.

[0111] It is easy to understand that after obtaining the coarse matching parameters, the coarse matching parameters are used to generate two infrared images and visible light images with the same viewing area, and then the FAST algorithm is used to extract features from the infrared image and visible light image, perform fine matching, and obtain the fine matching parameters.

[0112] In some embodiments, the using of the FAST algorithm to determine whether the pixel is a corner point includes:

[0113] Draw a circle with the pixel as the center and the preset radius, and calculate the grayscale values ​​of all the pixels that the circle passes through;

[0114] Determine in sequence whether the difference between the grayscale value of each pixel and the pixel is greater than or equal to a preset threshold, and obtain the target number of pixels greater than or equal to the preset threshold;

[0115] Determine whether the number of targets is greater than half of the total number of pixels. If the number of targets is greater than half of the total number of pixels, the pixel is regarded as a corner point.

[0116] It should be noted that the FAST algorithm is a commonly used corner feature extraction method in image registration. It has a fast extraction speed and the extracted feature points are widely distributed. For each pixel in the image, a circle is drawn with the pixel as the center. Among the pixels passing through the circumference, if the difference between the grayscale value of a certain number of consecutive pixels and the grayscale value of the center point is greater than or less than a certain threshold, then the center point is considered to be a corner point. Otherwise, the center point is not a corner point.

[0117] For example, for a pixel point P, its gray value is I p , the detection process of corner points is as follows:

[0118] (1) Draw a circle with the pixel as the center and 3 as the radius. The circumference of the circle passes through 16 pixels in total. The grayscale values ​​of these 16 pixels are I1, I2…, I 16 .

[0119] (2) First determine the difference between the grayscale value of the first pixel and the ninth pixel and the center pixel value. If I1-I p >t,I9-I p >t or I1-Ip <t, I9 - I p <t, then continue; if not satisfied, exclude this pixel point.

[0120] (3) Then judge the difference between the gray values of the 5th pixel point and the 13th pixel point and the central point pixel value. If it satisfies I5 - I p >t, I 13 - I p >t or I5 - I p <t, I 13 - I p <t, then continue; if not satisfied, exclude this pixel point.

[0121] (4) If the difference between the gray values of 9 or more consecutive pixel points among these 16 pixel points and the gray value of the central point is greater than or less than the preset threshold t, then the pixel point P is considered a corner point.

[0122] (5) Use the non - maximum suppression method to remove the dense points in the local area. By calculating the sum of the absolute values of the differences between the central point and the pixel points in the local area, retain the pixel point with the largest absolute value in the area.

[0123] In this embodiment, by taking this pixel point as the center of the circle, calculating the gray values of the pixel points inside the circle and comparing with the preset threshold, it is possible to quickly identify the pixel points with large gray value changes, thereby extracting corner points in the image, improving the accuracy and efficiency of corner detection, and reducing the calculation time and complexity of fine matching.

[0124] In this embodiment, by using the FAST algorithm to extract all corner points in the visible - light image and the infrared image, it is possible to effectively obtain the feature information in the image. Detect the corner points of the visible - light image and the infrared image respectively, obtain the visible - light corner point set and the infrared corner point set, extract more accurate features in the visible - light image and the infrared image, improve the accuracy of corner detection, achieve faster and more accurate feature alignment, and improve the efficiency and accuracy of the matching of infrared data and visible - light data.

[0125] In some embodiments, the fine matching of the visible - light corner point set and the infrared corner point set to obtain the fine - matching parameters includes:

[0126] Calculate the feature similarity of the visible - light corner point set and the infrared corner point set to obtain the initial paired feature points;

[0127] Based on the initial paired feature points, calculate the fundamental matrix;

[0128] Decompose the fundamental matrix to obtain the fine - matching parameters.

[0129] Figure 3This is a flow chart of the precise matching process provided by the embodiment of the present application, such as Figure 3 As shown in the figure, after obtaining the coarse matching parameters, the visible light image and the infrared image are generated and the corner points are calculated to obtain the visible light corner point set and the infrared corner point set. Feature similarity calculation and pairing are performed based on the obtained visible light corner point set and the infrared corner point set to obtain the initial paired feature points. Then, the basic matrix is ​​calculated and decomposed based on the initial paired feature points to obtain the fine matching parameters.

[0130] In this embodiment, by calculating the feature similarity of the visible light corner point set and the infrared corner point set, the initial pairing feature points are obtained, and the basic matrix is ​​calculated based on the initial pairing feature points. Then, the precise matching parameters are obtained by decomposing the basic matrix. This can eliminate errors caused by factors such as noise, illumination changes, or sensor differences, and realize the coarse-fine two-stage registration of infrared images and visible light images, thereby improving the efficiency and accuracy of matching infrared data and visible light data and reducing registration errors.

[0131] In some embodiments, decomposing the basic matrix to obtain precise matching parameters includes:

[0132] a. Select the minimum feature points required by the basic matrix to solve the basic matrix;

[0133] b. Substitute all initial paired feature points into the obtained basic matrix, determine whether each initial paired feature point is within a preset error range, and take the initial paired feature points within the preset error range as inliers to obtain the number of inliers;

[0134] c. Repeat steps a-b until the number of inliers is greater than the preset number of inliers or the number of iterations reaches the preset number of iterations;

[0135] d. Decompose the basic matrix with the largest number of inliers to obtain the precise matching parameters.

[0136] Exemplarily, the flow chart for obtaining precise matching parameters is as follows:

[0137] (1) Select at least four feature points required to solve the basic matrix;

[0138] (2) Solve the basic matrix E based on the selected minimum feature points;

[0139] (3) Substitute all the initial paired feature points into the obtained basic matrix, consider the matching pairs within the preset error range as inliers, and count the number of inliers;

[0140] (4) Compare the parameters of the current basic matrix with the number of inliers under the previously obtained optimal basic matrix parameters. The one with the greater number of inliers is used as the new optimal basic matrix parameter. Record the optimal basic matrix parameter and the number of inliers at this time.

[0141] (5) Repeat (1) to (4) until the number of interior points under the basic matrix parameters at this time is greater than the preset number of interior points or the number of iterations reaches the preset number of iterations.

[0142] (6) Decompose the basic matrix with the largest number of inliers to obtain the precise matching parameters.

[0143] Figure 4 This is the second flow chart of the coarse-fine two-stage registration method based on visible light images and infrared images provided in the embodiment of the present application, such as Figure 4 As shown in the figure, the corresponding 3DGS model and point cloud data are generated based on the visible light data and infrared data. The 3DGS model is roughly matched according to the point cloud data to obtain the rough matching parameters. Then, the visible light image and infrared image are generated according to the rough matching parameters. The FAST algorithm is used for fine matching. Finally, the two 3DGS models are rendered and synthesized according to the fine matching parameters to obtain a fusion image. Figure 5 This is a matching fusion effect diagram provided by the embodiment of the present application, such as Figure 5 As shown, the rendering is generated based on the pairing of visible light data and infrared data.

[0144] In this embodiment, the basic matrix is ​​solved by selecting the minimum feature points required by the basic matrix, and all the initial paired feature points are substituted into the obtained basic matrix to determine whether it is within the preset error range. By repeated iterations and continuous optimization of the number of inliers, the basic matrix with the largest number of inliers is finally selected for decomposition to obtain the precise matching parameters, thereby improving the accuracy of the precise matching parameters, enhancing the efficiency and accuracy of matching between infrared data and visible light data, and reducing the registration error.

[0145] The coarse-fine two-stage registration method based on visible light images and infrared images provided in the embodiments of the present application can be performed by a coarse-fine two-stage registration device based on visible light images and infrared images. In the embodiments of the present application, the coarse-fine two-stage registration method based on visible light images and infrared images is performed by a coarse-fine two-stage registration device based on visible light images and infrared images as an example to illustrate the coarse-fine two-stage registration method based on visible light images and infrared images provided in the embodiments of the present application.

[0146] The present application also provides a coarse-fine two-stage registration device based on visible light images and infrared images, such as Figure 6 As shown, the coarse-fine two-stage registration device based on visible light images and infrared images includes: an acquisition module 610 , a first processing module 620 , a coarse matching module 630 , a second processing module 640 , a fine matching module 650 and a generation module 660 .

[0147] An acquisition module 610 is configured to acquire visible light data and infrared data, wherein the visible light data is acquired by acquiring a target scene through a visible light camera sensor, and the infrared data is acquired by acquiring a target scene through an infrared camera sensor;

[0148] A first processing module 620 is configured to establish a first 3DGS model based on the visible light data and generate visible light point cloud data, and to establish a second 3DGS model based on the infrared data and generate infrared point cloud data;

[0149] A coarse matching module 630 is configured to extract eigenvalues ​​of the visible light point cloud data and the infrared point cloud data using a KPConv network and perform coarse matching to obtain coarse matching parameters, wherein the coarse matching parameters include a scaling factor, a rotation matrix, and a translation vector;

[0150] A second processing module 640 is configured to generate a visible light image and an infrared image based on the coarse matching parameters, and extract all corner points in the visible light image and the infrared image using a FAST algorithm to obtain a visible light corner point set and an infrared corner point set;

[0151] A fine matching module 650 is configured to perform fine matching on the visible light corner point set and the infrared corner point set to obtain fine matching parameters, wherein the fine matching parameters include a scaling factor, a rotation matrix, and a translation vector;

[0152] The generating module 660 is configured to obtain the target image based on the precise matching parameters, where the target image is a fusion image of the optical image and the infrared image.

[0153] According to the coarse-fine two-stage registration method based on visible light images and infrared images provided in the embodiments of the present application, 3DGS modeling is performed by acquiring visible light data and infrared data, and visible light point cloud data and infrared point cloud data are generated respectively. The KPConv network is used to perform coarse matching on the point cloud data to obtain coarse matching parameters and generate visible light images and infrared images. The FAST algorithm is used to extract corner point sets in the visible light image and infrared image, and fine matching is performed on them to obtain fine matching parameters. This realizes coarse-fine two-stage registration of infrared images and visible light images, improves the efficiency and accuracy of matching of infrared data and visible light data, and reduces registration errors.

[0154] The coarse-fine two-stage registration device based on visible light image and infrared image provided in the embodiment of the present application can achieve Figures 1 to 5 To avoid repetition, the various processes implemented in the embodiment of the coarse-fine two-stage registration method based on visible light images and infrared images are not described here.

[0155] In some embodiments, as Figure 7As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, each process of the above-mentioned embodiment of the coarse-fine two-stage registration method based on visible light images and infrared images is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0156] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0157] An embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various processes of the above-mentioned embodiment of the coarse-fine two-stage registration method based on visible light images and infrared images, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0158] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0159] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned coarse-fine two-stage registration method based on visible light images and infrared images.

[0160] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0161] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, which are coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the coarse-fine two-stage registration method based on visible light images and infrared images, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0162] It should be understood that the chip mentioned in the embodiments of the present application can also be called a device-level chip, a device chip, a chip device, or an on-chip device chip, etc.

[0163] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD-ROM), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the coarse-fine two-stage registration method based on visible light images and infrared images of each embodiment of the present application.

[0165] In the description of this application, "first feature" and "second feature" may include one or more such features.

[0166] In the description of this application, “plurality” means two or more.

[0167] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0168] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0169] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and intent of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A coarse-fine two-stage registration method based on visible light images and infrared images, characterized in that: The method comprises: Obtaining visible light data and infrared data, wherein the visible light data is obtained by collecting the target scene using a visible light camera sensor, and the infrared data is obtained by collecting the target scene using an infrared camera sensor; Establishing a first 3DGS model based on the visible light data and generating visible light point cloud data, establishing a second 3DGS model based on the infrared data and generating infrared point cloud data; Using a KPConv network to extract eigenvalues ​​of the visible light point cloud data and the infrared point cloud data and perform coarse matching to obtain coarse matching parameters, wherein the coarse matching parameters include a first scaling factor, a first rotation matrix, and a first translation vector; generating a visible light image and an infrared image based on the coarse matching parameters, and extracting all corner points in the visible light image and the infrared image using the FAST algorithm to obtain a visible light corner point set and an infrared corner point set; Performing precise matching on the visible light corner point set and the infrared corner point set to obtain precise matching parameters, wherein the precise matching parameters include a first scaling factor, a second rotation matrix, and a second translation vector; Based on the precise matching parameters, the target image is obtained, where the target image is a fusion image of the optical image and the infrared image.

2. The coarse-fine two-stage registration method based on visible light images and infrared images according to claim 1, characterized in that: The KPConv network is used to extract the characteristic values ​​of the visible light point cloud data and the infrared point cloud data and perform rough matching to obtain rough matching parameters, including: Performing multi-level uniform downsampling on the visible light point cloud data and the infrared point cloud data based on a KPConv network to obtain a first sampling set and a second sampling set; Performing feature extraction on the first sampling set and the second sampling set to obtain a first feature set and a second feature set; Performing layer-by-layer feature encoding on the first feature set and the second feature set based on KPConv kernel point convolution and feature pyramid network to obtain a visible light point cloud feature point set and an infrared point cloud feature point set; Rough matching is performed based on the visible light point cloud feature point set and the infrared point cloud feature point set to obtain rough matching parameters.

3. The coarse-fine two-stage registration method based on visible light images and infrared images according to claim 2, characterized in that: The coarse matching is performed based on the visible light point cloud feature point set and the infrared point cloud feature point set to obtain coarse matching parameters, including: An objective function is established based on the visible light point cloud feature point set and the infrared point cloud feature point set. The expression of the objective function is as follows: Where E is the sum of the squared Euclidean distances between all matching point pairs in the visible light point cloud feature point set and the infrared point cloud feature point set, s is the scaling factor, R is the rotation matrix, t is the translation vector, and q is the translation vector. i is the i-th feature point in the visible light point cloud feature point set, p i is the i-th feature point in the infrared point cloud feature point set, N p is the number of feature points in the visible light point cloud feature point set; Solving the minimum solution of the objective function to obtain a transformation matrix; The transformation matrix is ​​used as a coarse matching parameter.

4. The coarse-fine two-stage registration method based on visible light images and infrared images according to claim 1, characterized in that: The FAST algorithm is used to extract all corner points in the visible light image and the infrared image to obtain a visible light corner point set and an infrared corner point set, including: Traverse all pixels in the visible light image, and use the FAST algorithm to determine whether each pixel is a corner point. All pixels that are corner points in the visible light image are regarded as the visible light corner point set. All pixels in the infrared image are traversed, and for each pixel, the FAST algorithm is used to determine whether the pixel is a corner point. All pixels that are corner points in the infrared image are regarded as the infrared corner point set.

5. The coarse-fine two-stage registration method based on visible light images and infrared images according to claim 4, characterized in that: The FAST algorithm is used to determine whether the pixel is a corner point, including: Draw a circle with the pixel as the center and the preset radius, and calculate the grayscale values ​​of all the pixels that the circle passes through; Determine in sequence whether the difference between the grayscale value of each pixel and the pixel is greater than or equal to a preset threshold, and obtain the target number of pixels greater than or equal to the preset threshold; Determine whether the number of targets is greater than half of the total number of pixels. If the number of targets is greater than half of the total number of pixels, the pixel is regarded as a corner point.

6. The coarse-fine two-stage registration method based on visible light images and infrared images according to claim 1, characterized in that: The performing precise matching on the visible light corner point set and the infrared corner point set to obtain precise matching parameters includes: Calculating feature similarities between the visible light corner point set and the infrared corner point set to obtain initial paired feature points; Calculating a basic matrix based on the initial paired feature points; The basic matrix is ​​decomposed to obtain precise matching parameters.

7. The coarse-fine two-stage registration method based on visible light images and infrared images according to claim 6, characterized in that: Decomposing the basic matrix to obtain precise matching parameters includes: a. Select the minimum feature points required by the basic matrix to solve the basic matrix; b. Substitute all initial paired feature points into the obtained basic matrix, determine whether each initial paired feature point is within a preset error range, and take the initial paired feature points within the preset error range as inliers to obtain the number of inliers; c. Repeat steps a-b until the number of inliers is greater than the preset number of inliers or the number of iterations reaches the preset number of iterations; d. Decompose the basic matrix with the largest number of inliers to obtain the precise matching parameters.

8. A coarse-fine two-stage registration device based on visible light images and infrared images, implemented using the coarse-fine two-stage registration method based on visible light images and infrared images according to any one of claims 1 to 7, characterized in that: The device comprises: An acquisition module, configured to acquire visible light data and infrared data, wherein the visible light data is acquired by acquiring a target scene through a visible light camera sensor, and the infrared data is acquired by acquiring a target scene through an infrared camera sensor; A first processing module is configured to establish a first 3DGS model based on the visible light data and generate visible light point cloud data, and to establish a second 3DGS model based on the infrared data and generate infrared point cloud data; A coarse matching module is used to extract eigenvalues ​​of the visible light point cloud data and the infrared point cloud data using a KPConv network and perform coarse matching to obtain coarse matching parameters, wherein the coarse matching parameters include a scaling factor, a rotation matrix, and a translation vector; A second processing module is configured to generate a visible light image and an infrared image based on the coarse matching parameters, and extract all corner points in the visible light image and the infrared image using a FAST algorithm to obtain a visible light corner point set and an infrared corner point set; A fine matching module, configured to perform fine matching on the visible light corner point set and the infrared corner point set to obtain fine matching parameters, wherein the fine matching parameters include a scaling factor, a rotation matrix, and a translation vector; A generating module is used to obtain the target image based on the precise matching parameters, where the target image is a fusion image of the optical image and the infrared image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the coarse-fine two-stage registration method based on visible light images and infrared images is implemented as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the coarse-fine two-stage registration method based on visible light images and infrared images is implemented as described in any one of claims 1 to 7.

Images