Coarse-fine two-stage registration method and device based on visible light image and infrared image

Abstract

The application discloses a coarse-fine two-stage registration method based on visible light images and infrared images, and belongs to the technical field of image processing. The method comprises the following steps: establishing a first 3DGS model based on visible light data and generating visible light point cloud data, and establishing a second 3DGS model based on infrared data and generating infrared point cloud data; extracting feature values of the visible light point cloud data and the infrared point cloud data by using a KPConv network and performing coarse matching to obtain coarse matching parameters, wherein the coarse matching parameters comprise a first scaling factor, a first rotation matrix and a first translation vector; generating visible light images and infrared images based on the coarse matching parameters, extracting all corner points in the visible light images and the infrared images by using a FAST algorithm to obtain a visible light corner point set and an infrared corner point set for fine matching, obtaining fine matching parameters, wherein the fine matching parameters comprise a first scaling factor, a second rotation matrix and a second translation vector, and obtaining target images based on the fine matching parameters, so that the matching degree of the visible light 3DGS model and the infrared 3DGS model is improved.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a coarse and fine two-stage registration method and apparatus based on visible light images and infrared images. Background Technology

[0002] With the rapid development of 3DGS (3D Geometric Surface) technology, it has demonstrated significant advantages in 3D modeling and scene reconstruction in fields such as aerial surveying, autonomous driving, and augmented reality. It is particularly adept at generating high-fidelity, highly detailed 3D models when dealing with complex lighting, transparent materials, and dynamic scenes. However, in aerial surveying scenarios involving multiple sensors (visible light and infrared cameras) from UAVs working collaboratively, achieving high-precision registration and fusion of cross-modal data remains a core challenge hindering the further application of 3DGS technology.

[0003] Existing infrared and visible light data registration methods mainly employ local alignment techniques based on feature matching and global optimization methods based on mutual information. The former performs matching by extracting local features such as edges and corners from cross-modal images, but in scenarios with significant differences in spectral response, the discriminative power of feature descriptors is insufficient, easily leading to mismatches and registration failures. The latter achieves alignment by maximizing the statistical dependency between two images. Although it is robust to global grayscale distribution, it is prone to registration deviations due to local non-rigid deformation when facing dynamic targets or complex geometric structures in large-scale aerial survey scenarios.

[0004] Furthermore, existing methods are mostly limited to two-dimensional image planar registration, failing to consider geometric consistency constraints in three-dimensional space. This results in misalignment in the spatial topology of the reconstructed visible light and infrared 3DGS models, making it difficult to directly fuse them into a unified cross-modal 3D representation. Existing methods also suffer from low matching accuracy and efficiency, as well as slow matching speed, when matching visible light and infrared 3DGS models. Summary of the Invention

[0005] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a coarse-fine two-stage registration method and apparatus based on visible light images and infrared images, which improves the fusion performance of 3DGS models built using multiple sensors.

[0006] In a first aspect, this application provides a coarse-to-fine two-stage registration method based on visible light images and infrared images, the method comprising:

[0007] Acquire visible light data and infrared data, wherein the visible light data is obtained by acquiring the target scene through a visible light camera sensor, and the infrared data is obtained by acquiring the target scene through an infrared camera sensor;

[0008] A first 3DGS model is established based on the visible light data and visible light point cloud data is generated; a second 3DGS model is established based on the infrared data and infrared point cloud data is generated.

[0009] The KPConv network is used to extract the feature values ​​of the visible light point cloud data and the infrared point cloud data and perform coarse matching to obtain coarse matching parameters, which include a first scaling factor, a first rotation matrix and a first translation vector.

[0010] Based on the coarse matching parameters, a visible light image and an infrared image are generated. The FAST algorithm is used to extract all corner points in the visible light image and the infrared image to obtain a set of visible light corner points and a set of infrared corner points.

[0011] The visible light corner point set and the infrared corner point set are finely matched to obtain fine matching parameters, which include a first scaling factor, a second rotation matrix and a second translation vector.

[0012] Based on the fine matching parameters, the target image is obtained, which is a fusion image of the light image and the infrared image.

[0013] According to one embodiment of this application, the step of extracting feature values ​​from the visible light point cloud data and the infrared point cloud data using a KPConv network and performing coarse matching to obtain coarse matching parameters includes:

[0014] Based on the KPConv network, the visible light point cloud data and the infrared point cloud data are downsampled in multiple layers to obtain the first sampling set and the second sampling set.

[0015] Feature extraction is performed on the first sampling set and the second sampling set to obtain a first feature set and a second feature set;

[0016] Based on KPConv kernel convolution and feature pyramid network, the first feature set and the second feature set are encoded layer by layer to obtain the visible light point cloud feature point set and the infrared point cloud feature point set.

[0017] Coarse matching parameters are obtained by performing coarse matching based on the visible light point cloud feature point set and the infrared point cloud feature point set.

[0018] According to one embodiment of this application, the coarse matching based on the visible light point cloud feature point set and the infrared point cloud feature point set to obtain coarse matching parameters includes:

[0019] An objective function is established based on the visible light point cloud feature point set and the infrared point cloud feature point set. The expression of the objective function is as follows:

[0020]

[0021] Where E is the sum of the squared Euclidean distances between all matching point pairs in the visible light point cloud feature point set and the infrared point cloud feature point set, s is the scaling factor, R is the rotation matrix, t is the translation vector, and q is the translation vector. i Let p be the i-th feature point in the set of visible light point cloud feature points. i Let N be the i-th feature point in the set of infrared point cloud feature points. p The number of feature points in the set of feature points in the visible light point cloud;

[0022] Solve for the minimum solution of the objective function to obtain the transformation matrix;

[0023] The transformation matrix is ​​used as a coarse matching parameter.

[0024] According to one embodiment of this application, the step of extracting all corner points from visible light and infrared images using the FAST algorithm to obtain a visible light corner point set and an infrared corner point set includes:

[0025] Traverse all pixels in the visible light image, and for each pixel, use the FAST algorithm to determine whether the pixel is a corner point. Collect all pixels in the visible light image that are corner points as the visible light corner point set.

[0026] Traverse all pixels in the infrared image. For each pixel, use the FAST algorithm to determine whether the pixel is a corner point. Collect all pixels in the infrared image that are corner points as the infrared corner point set.

[0027] According to one embodiment of this application, the step of using the FAST algorithm to determine whether the pixel is a corner point includes:

[0028] Draw a circle with the pixel as the center and the default value as the radius, and calculate the grayscale values ​​of all pixels that the circle passes through.

[0029] The difference between the gray value of each pixel and the value of that pixel is determined sequentially to see if it is greater than or equal to a preset threshold, and the number of target pixels that are greater than or equal to the preset threshold is obtained.

[0030] Determine if the number of targets is greater than half the total number of pixels. If the number of targets is greater than half the total number of pixels, then designate that pixel as a corner point.

[0031] According to one embodiment of this application, the step of performing fine matching on the visible light corner point set and the infrared corner point set to obtain fine matching parameters includes:

[0032] Calculate the feature similarity between the visible light corner point set and the infrared corner point set to obtain the initial paired feature points;

[0033] Based on the initial paired feature points, calculate the fundamental matrix;

[0034] The fundamental matrix is ​​decomposed to obtain the fine matching parameters.

[0035] According to one embodiment of this application, the decomposition of the fundamental matrix to obtain the fine matching parameters includes:

[0036] a. Solve for the fundamental matrix by selecting the minimum number of feature points required for the fundamental matrix;

[0037] b. Substitute all the initial paired feature points into the derived fundamental matrix and determine whether each initial paired feature point is within the preset error range. Select the initial paired feature points within the preset error range as interior points and obtain the number of interior points. c. Repeat steps a-b until the number of interior points is greater than the preset number of interior points or the number of iterations reaches the preset number of iterations. d. Decompose the fundamental matrix with the largest number of interior points to obtain the fine matching parameters.

[0038] Secondly, this application provides a coarse and fine two-stage registration device based on visible light images and infrared images, the device comprising:

[0039] The acquisition module is used to acquire visible light data and infrared data. The visible light data is obtained by acquiring the target scene through a visible light camera sensor, and the infrared data is obtained by acquiring the target scene through an infrared camera sensor.

[0040] The first processing module is used to establish a first 3DGS model based on the visible light data and generate visible light point cloud data, and to establish a second 3DGS model based on the infrared data and generate infrared point cloud data.

[0041] The coarse matching module is used to extract the feature values ​​of the visible light point cloud data and the infrared point cloud data using the KPConv network and perform coarse matching to obtain coarse matching parameters, which include scaling factor, rotation matrix and translation vector.

[0042] The second processing module is used to generate visible light images and infrared images based on the coarse matching parameters, and to extract all corner points in the visible light images and infrared images using the FAST algorithm to obtain visible light corner point sets and infrared corner point sets.

[0043] The fine matching module is used to perform fine matching on the visible light corner point set and the infrared corner point set to obtain fine matching parameters, which include scaling factor, rotation matrix and translation vector.

[0044] A generation module is used to obtain the target image based on the fine matching parameters, wherein the target image is a fusion image of the light image and the infrared image.

[0045] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the coarse and fine two-stage registration method based on visible light images and infrared images as described in the first aspect above.

[0046] Fourthly, this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the coarse-fine two-stage registration method based on visible light images and infrared images as described in the first aspect above.

[0047] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the coarse and fine two-stage registration method based on visible light images and infrared images as described in the first aspect.

[0048] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the coarse and fine two-stage registration method based on visible light images and infrared images as described in the first aspect above.

[0049] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application.

[0050] The present invention provides a coarse and fine two-stage registration method based on visible light images and infrared images, which has the following advantages over the prior art:

[0051] (1) This invention acquires visible light data and infrared data to perform 3DGS modeling, and generates visible light point cloud data and infrared point cloud data respectively. The KPConv network is used to perform coarse matching on the point cloud data to obtain coarse matching parameters and generate visible light images and infrared images. The FAST algorithm is used to extract the corner point set in the visible light image and infrared image, and fine matching is performed on it to obtain fine matching parameters. This realizes the coarse-fine two-stage registration of infrared images and visible light images, improves the efficiency and accuracy of matching infrared data and visible light data, and reduces registration error.

[0052] (2) This invention performs multi-level uniform downsampling on visible light point cloud data and infrared point cloud data based on KPConv network to obtain a first sampling set and a second sampling set, and then performs feature extraction to obtain a first feature set and a second feature set. This can extract more accurate and efficient feature information. By combining KPConv kernel convolution and feature pyramid network to encode the feature set layer by layer, the expressive power of the features is further improved. Through the coarse matching process, the time and complexity of subsequent matching can be reduced, and the efficiency and accuracy of matching infrared data and visible light data are improved.

[0053] (3) This invention uses the FAST algorithm to extract all corner points in visible light images and infrared images, which can effectively obtain feature information in the images. Corner point detection is performed on visible light images and infrared images respectively to obtain visible light corner point sets and infrared corner point sets. More accurate features can be extracted from visible light images and infrared images, improving the accuracy of corner point detection, achieving faster and more accurate feature alignment, and improving the efficiency and accuracy of matching infrared data and visible light data. Attached Figure Description

[0054] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0055] Figure 1 This is one of the flowcharts of the coarse and fine two-stage registration method based on visible light images and infrared images provided in the embodiments of this application;

[0056] Figure 2 This is a schematic diagram of the coarse matching process provided in the embodiments of this application;

[0057] Figure 3 This is a schematic diagram of the fine matching process provided in the embodiments of this application;

[0058] Figure 4 This is the second flowchart of the coarse and fine two-stage registration method based on visible light images and infrared images provided in the embodiments of this application;

[0059] Figure 5 This is a matching and fusion effect diagram provided in the embodiments of this application;

[0060] Figure 6 This is a schematic diagram of the coarse and fine two-stage registration device based on visible light images and infrared images provided in the embodiments of this application;

[0061] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0063] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0064] The following description, in conjunction with the accompanying drawings, details the coarse-fine two-stage registration method based on visible light images and infrared images, the coarse-fine two-stage registration device based on visible light images and infrared images, the electronic device, and the readable storage medium provided in this application, through specific embodiments and application scenarios.

[0065] Among them, the coarse and fine two-stage registration method based on visible light images and infrared images can be applied to the terminal, and can be executed by the hardware or software in the terminal.

[0066] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0067] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0068] The coarse and fine two-stage registration method based on visible light images and infrared images provided in this application embodiment can be implemented by an electronic device or a functional module or entity in an electronic device that can implement the coarse and fine two-stage registration method based on visible light images and infrared images. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablet computers, computers, cameras and wearable devices. The following uses an electronic device as the implementation subject to illustrate the coarse and fine two-stage registration method based on visible light images and infrared images provided in this application embodiment.

[0069] Figure 1 This is one of the flowcharts illustrating the coarse and fine two-stage registration method based on visible light and infrared images provided in this application embodiment, such as... Figure 1 As shown, the coarse and fine two-stage registration method based on visible light images and infrared images includes steps 110, 120, 130, 140, 150 and 160.

[0070] Step 110: Acquire visible light data and infrared data. The visible light data is obtained by acquiring the target scene through a visible light camera sensor, and the infrared data is obtained by acquiring the target scene through an infrared camera sensor.

[0071] It is easy to understand that the visible light camera sensor and the infrared camera sensor respectively collect video or image data of the target scene to obtain visible light data and infrared data. The visible light data is image or video data in the visible light band, and the infrared data is image or video data in the infrared band.

[0072] Step 120: Based on the visible light data, establish a first 3DGS model and generate visible light point cloud data; based on the infrared data, establish a second 3DGS model and generate infrared point cloud data.

[0073] It should be noted that the visible light data is reconstructed in three dimensions to generate the first 3DGS (3D Geometric Surface) model. The three-dimensional coordinates are calculated using stereo matching and triangulation techniques to generate visible light point cloud data. The infrared data is reconstructed in three dimensions to generate the second 3DGS model. The temperature information in the infrared data is mapped to three-dimensional space to generate infrared point cloud data.

[0074] Step 130: Use the KPConv network to extract the feature values ​​of the visible light point cloud data and the infrared point cloud data and perform coarse matching to obtain coarse matching parameters. The coarse matching parameters include a first scaling factor, a first rotation matrix and a first translation vector.

[0075] It is easy to understand that KPConv (KernelPoint Convolution) is used to extract features from visible light point cloud data and infrared point cloud data, obtain the feature values ​​of visible light point cloud data and infrared point cloud data, and perform coarse matching to obtain coarse matching parameters. The coarse matching parameters include the first scaling factor, the first rotation matrix and the first translation vector.

[0076] The first scaling factor represents the size difference between the visible light point cloud data and the infrared point cloud data, that is, how to scale one point cloud data to match the other point cloud data. The first rotation matrix represents the rotation relationship between the visible light point cloud data and the infrared point cloud data, that is, how to rotate one point cloud data to align it with the other point cloud data. The first translation vector represents the translation relationship between the visible light point cloud data and the infrared point cloud data.

[0077] Step 140: Generate a visible light image and an infrared image based on the coarse matching parameters, and use the FAST algorithm to extract all corner points in the visible light image and the infrared image to obtain a visible light corner point set and an infrared corner point set;

[0078] Furthermore, based on the coarse matching parameters, an image generation method is used to convert the visible light data and infrared data into corresponding image formats, resulting in visible light images and infrared images.

[0079] The FAST (Features from Accelerated Segment Test) algorithm is an algorithm used to detect corner points (or key points) in images. Corner points are points in an image that have significant features. These points usually have large variations in local areas of the image (such as edge intersections or locations with obvious texture changes). The FAST algorithm extracts all corner points in visible light and infrared images to obtain visible light corner point sets and infrared corner point sets.

[0080] Step 150: Perform fine matching on the visible light corner point set and the infrared corner point set to obtain fine matching parameters, the fine matching parameters including a first scaling factor, a second rotation matrix and a second translation vector;

[0081] Finally, in order to further improve the matching between visible light data and infrared data, fine matching is performed based on the visible light corner point set and the infrared corner point set to obtain fine matching parameters, which include a first scaling factor, a second rotation matrix and a second translation vector.

[0082] It is worth noting that the first scaling factor in the fine matching parameters is the same as the first scaling factor in the coarse matching parameters.

[0083] Step 160: Based on the fine matching parameters, obtain the target image, which is the fusion image of the light image and the infrared image.

[0084] According to the coarse-fine two-stage registration method based on visible light and infrared images provided in the embodiments of this application, 3DGS modeling is performed by acquiring visible light and infrared data, and visible light point cloud data and infrared point cloud data are generated respectively. The KPConv network is used to perform coarse matching on the point cloud data to obtain coarse matching parameters and generate visible light and infrared images. The FAST algorithm is used to extract the corner point set in the visible light and infrared images, and fine matching is performed on them to obtain fine matching parameters. This realizes the coarse-fine two-stage registration of infrared and visible light images, improves the efficiency and accuracy of matching infrared and visible light data, and reduces registration errors.

[0085] In some embodiments, the step of extracting feature values ​​from the visible light point cloud data and the infrared point cloud data using a KPConv network and performing coarse matching to obtain coarse matching parameters includes:

[0086] Based on the KPConv network, the visible light point cloud data and the infrared point cloud data are downsampled in multiple layers to obtain the first sampling set and the second sampling set.

[0087] Feature extraction is performed on the first sampling set and the second sampling set to obtain a first feature set and a second feature set;

[0088] Based on KPConv kernel convolution and feature pyramid network, the first feature set and the second feature set are encoded layer by layer to obtain the visible light point cloud feature point set and the infrared point cloud feature point set.

[0089] Coarse matching parameters are obtained by performing coarse matching based on the visible light point cloud feature point set and the infrared point cloud feature point set.

[0090] Figure 2 This is a schematic diagram of the coarse matching process provided in the embodiments of this application, such as... Figure 2 As shown, firstly, the KPConv network is used for multi-level downsampling to extract point-by-point initial features. After obtaining point cloud features at different scales, feature matching is performed. Based on the matched point cloud pairs, coarse matching parameters for visible light point cloud data and infrared point cloud data are calculated. The specific process is as follows:

[0091] (1) Point Cloud Downsampling and Feature Encoding: Multi-level uniform downsampling is implemented based on the KPConv network. For the input visible light point cloud data and infrared point cloud data, a point X is selected as the center of a sphere, and a sphere with radius r is defined. Several core points x are selected within the sphere, each with a weight matrix. For any point within the sphere, the weight matrix of that point is calculated using a kernel function, and the feature of that point is transformed using this matrix. For each point within the sphere, new features are calculated using the above method. Finally, these features are accumulated to obtain the feature of point X. The formula is as follows:

[0092]

[0093] Wherein, g(x) i -x) is the kernel function, f i F represents the features after convolution. * (g(x)) is a characteristic of point x.

[0094] The formula for calculating the kernel function is shown below:

[0095]

[0096] in, W represents the position of the kth core point. k As the first weighting coefficient, The second weighting coefficient is calculated using the following formula:

[0097]

[0098] (2) Within each point cloud scale, KPConv kernel convolution and a feature pyramid network are used to implement layer-by-layer feature encoding of the point cloud, obtaining point cloud features at different sampling scales. Finally, visible light point cloud feature points are obtained after N (e.g., 5) downsampling operations. and infrared point cloud feature points

[0099] In this embodiment, visible light point cloud data and infrared point cloud data are downsampled at multiple levels using a KPConv network to obtain a first sampling set and a second sampling set. Feature extraction is then performed to obtain a first feature set and a second feature set, which can extract more accurate and efficient feature information. By combining KPConv kernel convolution and feature pyramid network to encode the feature sets layer by layer, the expressive power of the features is further improved. Through the coarse matching process, the time and complexity of subsequent matching can be reduced, thereby improving the efficiency and accuracy of matching infrared data and visible light data.

[0100] In some embodiments, the coarse matching based on the visible light point cloud feature point set and the infrared point cloud feature point set to obtain coarse matching parameters includes:

[0101] An objective function is established based on the visible light point cloud feature point set and the infrared point cloud feature point set. The expression of the objective function is as follows:

[0102]

[0103] Where E is the sum of the squared Euclidean distances between all matching point pairs in the visible light point cloud feature point set and the infrared point cloud feature point set, s is the scaling factor, R is the rotation matrix, t is the translation vector, and q is the translation vector. i Let p be the i-th feature point in the set of visible light point cloud feature points. i Let N be the i-th feature point in the set of infrared point cloud feature points. p The number of feature points in the set of feature points in the visible light point cloud;

[0104] Solve for the minimum solution of the objective function to obtain the transformation matrix;

[0105] The transformation matrix is ​​used as a coarse matching parameter.

[0106] It is easy to understand that after obtaining the relevant matching feature points of visible light point clouds and infrared point clouds at different scales, the transformation relationship {s, R, t} between the two point clouds is solved, where s is the scaling factor, R is the rotation matrix, and t is the translation vector. This problem can be transformed into calculating the set of two matching feature points P = {p1, p2, ..., p...}. n} and Q = {q1,q2,…,q n The solution {s, R, t} that minimizes the objective function is the sum of the squared Euclidean distances between all matching point pairs in the visible light point cloud feature point set and the infrared point cloud feature point set. The calculated transformation matrix is ​​then used as the coarse matching parameter.

[0107] In this embodiment, by establishing an objective function, the geometric relationship between visible light and infrared point cloud data is quantified, and the minimum solution of the objective function is obtained to obtain the transformation matrix, which can better describe the transformation relationship between visible light point cloud data and infrared point cloud data. This achieves coarse matching between infrared data and visible light data, reduces errors caused by differences between different sensors and data noise, and improves the robustness and efficiency of registration.

[0108] In some embodiments, the step of extracting all corner points from visible light and infrared images using the FAST algorithm to obtain visible light corner point sets and infrared corner point sets includes:

[0109] Traverse all pixels in the visible light image, and for each pixel, use the FAST algorithm to determine whether the pixel is a corner point. Collect all pixels in the visible light image that are corner points as the visible light corner point set.

[0110] Traverse all pixels in the infrared image. For each pixel, use the FAST algorithm to determine whether the pixel is a corner point. Collect all pixels in the infrared image that are corner points as the infrared corner point set.

[0111] It is easy to understand that after obtaining the coarse matching parameters, two infrared and visible light images with the same viewing area are generated using the coarse matching parameters. Then, the FAST algorithm is used to extract features from the infrared and visible light images and perform fine matching to obtain the fine matching parameters.

[0112] In some embodiments, determining whether a pixel is a corner point using the FAST algorithm includes:

[0113] Draw a circle with the pixel as the center and the default value as the radius, and calculate the grayscale values ​​of all pixels that the circle passes through.

[0114] The difference between the gray value of each pixel and the value of that pixel is determined sequentially to see if it is greater than or equal to a preset threshold, and the number of target pixels that are greater than or equal to the preset threshold is obtained.

[0115] Determine if the number of targets is greater than half the total number of pixels. If the number of targets is greater than half the total number of pixels, then designate that pixel as a corner point.

[0116] It should be noted that the FAST algorithm is a commonly used corner feature extraction method in image registration. It is fast and the extracted feature points are widely distributed. For each pixel in the image, a circle is drawn with the pixel as the center. If a certain number of consecutive pixels on the circumference have a gray value difference greater than or less than a certain threshold with the gray value of the center point, then the center point is considered to be a corner point; otherwise, the center point is not a corner point.

[0117] For example, for a pixel P, its grayscale value is I. p The corner detection process is as follows:

[0118] (1) Draw a circle with the pixel as the center and a radius of 3. The circle passes through 16 pixels, and the gray values ​​of these 16 pixels are I1, I2, ..., I... 16 .

[0119] (2) First, determine the difference between the gray values ​​of the 1st and 9th pixels and the center pixel value. If the difference is satisfied, then... p >t,I9-I p >t or I1-Ip <t, I9 - I p <t, then continue; if not satisfied, exclude this pixel point.

[0120] (3) Then, determine the differences between the gray values of the 5th pixel point and the 13th pixel point and the gray value of the central pixel point. If it satisfies I5 - I p >t, I 13 - I p >t or I5 - I p <t, I 13 - I p <t, then continue; if not satisfied, exclude this pixel point.

[0121] (4) If the differences between the gray values of 9 or more consecutive pixel points among these 16 pixel points and the gray value of the central pixel point are greater than or less than the preset threshold t, then the pixel point P is considered a corner point.

[0122] (5) Use the non - maximum suppression method to remove the dense points in the local area. By calculating the sum of the absolute values of the differences between the central point and the pixel points in the local area, retain the pixel point with the largest absolute value in the area.

[0123] In this embodiment, by taking this pixel point as the center of a circle, calculating the gray values of the pixel points within the circle and comparing them with the preset threshold, it is possible to quickly identify the pixel points with large gray - value changes, thereby extracting corner points in the image, improving the accuracy and efficiency of corner detection, and reducing the calculation time and complexity of fine matching.

[0124] In this embodiment, by using the FAST algorithm to extract all corner points in the visible - light image and the infrared image, it is possible to effectively obtain the feature information in the images. Detect the corner points of the visible - light image and the infrared image respectively to obtain the visible - light corner - point set and the infrared corner - point set, be able to extract more accurate features in the visible - light image and the infrared image, improve the accuracy of corner detection, achieve faster and more accurate feature alignment, and enhance the efficiency and accuracy of the matching between infrared data and visible - light data.

[0125] In some embodiments, the fine - matching of the visible - light corner - point set and the infrared corner - point set to obtain the fine - matching parameters includes:

[0126] Calculate the feature similarity of the visible - light corner - point set and the infrared corner - point set to obtain the initial paired feature points;

[0127] Based on the initial paired feature points, calculate the fundamental matrix;

[0128] Decompose the fundamental matrix to obtain the fine - matching parameters.

[0129] Figure 3This is a schematic diagram of the fine matching process provided in the embodiments of this application, such as... Figure 3 As shown, after obtaining the coarse matching parameters, visible light images and infrared images are generated and corner points are calculated to obtain visible light corner point sets and infrared corner point sets. Based on the obtained visible light corner point sets and infrared corner point sets, feature similarity is calculated and paired to obtain initial paired feature points. Then, based on the initial paired feature points, the fundamental matrix is ​​calculated and decomposed to obtain fine matching parameters.

[0130] In this embodiment, by calculating the feature similarity between the visible light corner point set and the infrared corner point set, initial paired feature points are obtained, and a fundamental matrix is ​​calculated based on the initial paired feature points. Then, by decomposing the fundamental matrix, fine matching parameters are obtained. This can eliminate errors caused by factors such as noise, illumination changes, or sensor differences, and realize coarse-fine two-stage registration of infrared and visible light images, improving the efficiency and accuracy of matching infrared and visible light data and reducing registration errors.

[0131] In some embodiments, the decomposition of the fundamental matrix to obtain the fine matching parameters includes:

[0132] a. Solve for the fundamental matrix by selecting the minimum number of feature points required for the fundamental matrix;

[0133] b. Substitute all the initial paired feature points into the derived fundamental matrix, determine whether each initial paired feature point is within the preset error range, and take the initial paired feature points within the preset error range as interior points to obtain the number of interior points;

[0134] c. Repeat steps a-b until the number of interior points is greater than the preset number of interior points or the number of iterations reaches the preset number of iterations;

[0135] d. Decompose the fundamental matrix with the largest number of internal points to obtain the fine matching parameters.

[0136] An example flowchart for obtaining the fine-match parameters is shown below:

[0137] (1) Select the minimum four feature points required to solve the fundamental matrix;

[0138] (2) Solve for the fundamental matrix E based on the minimum number of selected feature points;

[0139] (3) Substitute all the initial paired feature points into the derived fundamental matrix. Matching pairs within the preset error range are considered interior points, and the number of interior points is counted.

[0140] (4) Compare the parameters of the current base matrix with the number of interior points under the parameters of the previous optimal base matrix. The one with more interior points is taken as the new optimal base matrix parameter. Record the optimal base matrix parameter and the number of interior points at this time.

[0141] (5) Repeat (1)-(4) until the number of interior points under the parameters of the base matrix is ​​greater than the preset number of interior points or the number of iterations reaches the preset number of iterations.

[0142] (6) Decompose the fundamental matrix with the largest number of internal points to obtain the fine matching parameters.

[0143] Figure 4 This is the second schematic flowchart of the coarse and fine two-stage registration method based on visible light images and infrared images provided in the embodiments of this application, as shown below. Figure 4 As shown, corresponding 3DGS models and point cloud data are generated based on visible light and infrared data. Coarse matching is performed on the 3DGS models using the point cloud data to obtain coarse matching parameters. Then, visible light and infrared images are generated based on these parameters, and fine matching is performed using the FAST algorithm. Finally, the two 3DGS models are rendered and synthesized based on the fine matching parameters to obtain a fused image. Figure 5 This is a matching and fusion effect diagram provided in the embodiments of this application, such as... Figure 5 As shown, this rendering was generated based on the pairing of visible light data and infrared data.

[0144] In this embodiment, the fundamental matrix is ​​solved by selecting the minimum number of feature points required for the fundamental matrix, and all initial paired feature points are substituted into the obtained fundamental matrix to determine whether they are within the preset error range. By iterating repeatedly and continuously optimizing the number of interior points, the fundamental matrix with the most interior points is finally selected for decomposition to obtain the fine matching parameters. This improves the accuracy of the fine matching parameters, enhances the efficiency and accuracy of matching infrared and visible light data, and reduces registration errors.

[0145] The coarse-fine two-stage registration method based on visible light and infrared images provided in this application can be implemented by a coarse-fine two-stage registration device based on visible light and infrared images. This application uses the example of a coarse-fine two-stage registration device based on visible light and infrared images executing the coarse-fine two-stage registration method to illustrate the coarse-fine two-stage registration device based on visible light and infrared images provided in this application.

[0146] This application also provides a coarse-fine two-stage registration device based on visible light images and infrared images, such as... Figure 6 As shown, the coarse and fine two-stage registration device based on visible light images and infrared images includes: an acquisition module 610, a first processing module 620, a coarse matching module 630, a second processing module 640, a fine matching module 650, and a generation module 660.

[0147] The acquisition module 610 is used to acquire visible light data and infrared data. The visible light data is obtained by acquiring the target scene through a visible light camera sensor, and the infrared data is obtained by acquiring the target scene through an infrared camera sensor.

[0148] The first processing module 620 is used to establish a first 3DGS model based on the visible light data and generate visible light point cloud data, and to establish a second 3DGS model based on the infrared data and generate infrared point cloud data.

[0149] The coarse matching module 630 is used to extract the feature values ​​of the visible light point cloud data and the infrared point cloud data using the KPConv network and perform coarse matching to obtain coarse matching parameters, which include scaling factor, rotation matrix and translation vector.

[0150] The second processing module 640 is used to generate a visible light image and an infrared image based on the coarse matching parameters, and to extract all corner points in the visible light image and the infrared image using the FAST algorithm to obtain a set of visible light corner points and a set of infrared corner points.

[0151] The fine matching module 650 is used to perform fine matching on the visible light corner point set and the infrared corner point set to obtain fine matching parameters, which include scaling factor, rotation matrix and translation vector.

[0152] The generation module 660 is used to obtain the target image based on the fine matching parameters, wherein the target image is a fusion image of the light image and the infrared image.

[0153] According to the coarse-fine two-stage registration method based on visible light and infrared images provided in the embodiments of this application, 3DGS modeling is performed by acquiring visible light and infrared data, and visible light point cloud data and infrared point cloud data are generated respectively. The KPConv network is used to perform coarse matching on the point cloud data to obtain coarse matching parameters and generate visible light and infrared images. The FAST algorithm is used to extract the corner point set in the visible light and infrared images, and fine matching is performed on them to obtain fine matching parameters. This realizes the coarse-fine two-stage registration of infrared and visible light images, improves the efficiency and accuracy of matching infrared and visible light data, and reduces registration errors.

[0154] The coarse and fine two-stage registration device based on visible light and infrared images provided in this application embodiment can achieve... Figures 1 to 5 The various processes implemented in the embodiment of the coarse and fine two-stage registration method based on visible light images and infrared images will not be described again here to avoid repetition.

[0155] In some embodiments, such as Figure 7As shown, this application embodiment also provides an electronic device 700, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, it implements the various processes of the above-described coarse and fine two-stage registration method embodiment based on visible light images and infrared images, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0156] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0157] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described coarse and fine two-stage registration method based on visible light images and infrared images, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0158] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0159] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described coarse and fine two-stage registration method based on visible light images and infrared images.

[0160] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0161] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described coarse and fine two-stage registration method based on visible light images and infrared images, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0162] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a device-level chip, device chip, chip device, or on-chip device chip, etc.

[0163] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the coarse and fine two-stage registration method based on visible light images and infrared images of the various embodiments of this application.

[0165] In the description of this application, "first feature" and "second feature" may include one or more of the features.

[0166] In the description of this application, "multiple" means two or more.

[0167] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0168] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0169] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A coarse-fine two-stage registration method based on visible light images and infrared images, characterized in that, The method comprises: acquiring visible light data and infrared data, the visible light data being obtained by collecting a target scene through a visible light camera sensor, and the infrared data being obtained by collecting the target scene through an infrared camera sensor; establishing a first 3DGS model based on the visible light data and generating visible light point cloud data, and establishing a second 3DGS model based on the infrared data and generating infrared point cloud data; extracting feature values of the visible light point cloud data and the infrared point cloud data by using a KPConv network and performing coarse matching to obtain coarse matching parameters, the coarse matching parameters comprising a first scaling factor, a first rotation matrix and a first translation vector; generating a visible light image and an infrared image based on the coarse matching parameters, and extracting all corner points in the visible light image and the infrared image by using a FAST algorithm to obtain a visible light corner point set and an infrared corner point set; performing fine matching on the visible light corner point set and the infrared corner point set to obtain fine matching parameters, the fine matching parameters comprising a first scaling factor, a second rotation matrix and a second translation vector; based on the fine matching parameters, obtaining a target image, the target image being a fusion image of the visible light image and the infrared image; the fine matching on the visible light corner point set and the infrared corner point set to obtain fine matching parameters comprises: calculating feature similarity of the visible light corner point set and the infrared corner point set to obtain initial paired feature points; calculating a fundamental matrix based on the initial paired feature points; decomposing the fundamental matrix to obtain fine matching parameters.

2. The coarse-fine two-stage registration method based on visible and infrared images according to claim 1, characterized in that, the extracting feature values of the visible light point cloud data and the infrared point cloud data by using a KPConv network and performing coarse matching to obtain coarse matching parameters comprises: performing multi-level uniform down-sampling on the visible light point cloud data and the infrared point cloud data based on a KPConv network to obtain a first sampling set and a second sampling set; performing feature extraction on the first sampling set and the second sampling set to obtain a first feature set and a second feature set; performing layer-by-layer feature encoding on the first feature set and the second feature set based on a KPConv kernel point convolution and a feature pyramid network to obtain a visible light point cloud feature point set and an infrared point cloud feature point set; performing coarse matching based on the visible light point cloud feature point set and the infrared point cloud feature point set to obtain coarse matching parameters.

3. The coarse-fine two-stage registration method based on visible and infrared images according to claim 2, characterized in that, the performing coarse matching based on the visible light point cloud feature point set and the infrared point cloud feature point set to obtain coarse matching parameters comprises: establishing an objective function based on the visible light point cloud feature point set and the infrared point cloud feature point set, an expression of the objective function being as follows: ； wherein E is the sum of the square of the Euclidean distance between all matching point pairs in the visible light point cloud feature point set and the infrared point cloud feature point set, s is a scaling factor, R is a rotation matrix, and t is a translation vector, is the i-th feature point in the visible light point cloud feature point set, is the i-th feature point in the infrared point cloud feature point set, is the number of feature points in the visible light point cloud feature point set; solving a minimum solution of the objective function to obtain a conversion matrix; taking the conversion matrix as the coarse matching parameters. 4.The coarse-fine two-stage registration method based on visible light images and infrared images according to claim 1, characterized in that, the extracting all corner points in the visible light image and the infrared image by using a FAST algorithm to obtain a visible light corner point set and an infrared corner point set comprises: traversing all pixel points in the visible light image, and for each pixel point, determining whether the pixel point is a corner point by using a FAST algorithm, and taking all pixel points that are corner points in the visible light image as the visible light corner point set; Traverse all pixel points in the infrared image, for each pixel point, determine whether the pixel point is a corner point by using a FAST algorithm, and take all pixel points that are corner points in the infrared image as an infrared corner point set.

5. The coarse-fine two-stage registration method based on visible and infrared images according to claim 4, characterized in that, The determining whether the pixel point is a corner point by using the FAST algorithm comprises: drawing a circle with the pixel point as a center and a preset value as a radius, and calculating gray values of all pixel points passed by the circle; sequentially determining whether a difference between a gray value of each pixel point and the pixel point is greater than or equal to a preset threshold value, to obtain a target number of pixel points greater than or equal to the preset threshold value; determining whether the target number is greater than one half of a number of all pixel points, and taking the pixel point as a corner point if the target number is greater than one half of the number of all pixel points. 6.The coarse-fine two-stage registration method based on visible light images and infrared images according to claim 1, wherein, The decomposing the basis matrix to obtain the precise matching parameter comprises: a. selecting a minimum number of feature points required by the basis matrix to solve the basis matrix; b. substituting all initial matching feature points into the solved basis matrix, determining whether each initial matching feature point is within a preset error range, taking the initial matching feature points within the preset error range as inliers, and obtaining a number of inliers; c. repeating steps a-b until the number of inliers is greater than a preset number of inliers or an iteration number reaches a preset iteration number; d. decomposing the basis matrix with the maximum number of inliers to obtain the precise matching parameter.

7. A coarse-fine two-stage registration apparatus based on visible and infrared images, implemented by the coarse-fine two-stage registration method based on visible and infrared images according to any one of claims 1 to 6, characterized in that, The device comprises: an acquisition module configured to acquire visible light data and infrared data, the visible light data being obtained by collecting a target scene through a visible light camera sensor, and the infrared data being obtained by collecting the target scene through an infrared camera sensor; a first processing module configured to establish a first 3DGS model based on the visible light data and generate visible light point cloud data, and establish a second 3DGS model based on the infrared data and generate infrared point cloud data; a coarse matching module configured to extract feature values of the visible light point cloud data and the infrared point cloud data by using a KPConv network and perform coarse matching to obtain coarse matching parameters, the coarse matching parameters comprising a scaling factor, a rotation matrix, and a translation vector; a second processing module configured to generate a visible light image and an infrared image based on the coarse matching parameters, extract all corner points in the visible light image and the infrared image by using a FAST algorithm, and obtain a visible light corner point set and an infrared corner point set; a precise matching module configured to perform precise matching on the visible light corner point set and the infrared corner point set to obtain precise matching parameters, the precise matching parameters comprising a scaling factor, a rotation matrix, and a translation vector; a generation module configured to obtain the target image based on the precise matching parameters, the target image being a fusion image of the visible light image and the infrared image.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the coarse-precise two-stage registration method based on the visible light image and the infrared image according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the coarse-precise two-stage registration method based on the visible light image and the infrared image according to any one of claims 1 to 6.

Images