RCF and SuperRetina combined edge detection feature matching image registration method, system and device and storage medium

Abstract

The invention discloses an RCF and SuperRetina combined edge detection feature matching image registration method, system and device and a storage medium, and is applied to the technical field of image processing, and the method comprises the steps: carrying out the initial registration processing of collected visible light and infrared images through employing initial registration parameters; performing infrared and visible light image edge extraction and feature key point detection and description on the dual-light image after initial registration processing by using an edge extraction feature detection model combining RCF and SuperRetina; according to the method, a key point and a description point are obtained, a BFMatcher method is utilized to match the key point and the description point, an affine transformation homography matrix between matching features is calculated by utilizing a findHomographic method based on the point with the best matching correlation, and final registration is carried out on an infrared initial registration image by utilizing a warpPerspective method according to the matrix. According to the invention, the accuracy and reliability of image registration are obviously improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image registration method, system, device and storage medium combining RCF and SuperRetina edge detection feature matching. Background Art

[0002] In today's multimodal image information processing field, the fusion technology of infrared images and visible light images occupies an important position. Visible light images, with their clear presentation of surface texture and color of objects, conform to human visual perception habits and provide rich and intuitive information in object recognition and scene understanding. Infrared images are based on the thermal radiation characteristics of objects, and can effectively detect target objects at night, in low light or in bad weather conditions, making up for the shortcomings of visible light images in these environments. Accurately aligning and fusing infrared images with visible light images has shown great application potential in many fields. In the field of security monitoring, all-weather and all-round monitoring coverage can be achieved, and the fused images can clearly identify the target whether it is in a well-lit day or a dark night. In the autonomous driving scenario, infrared images can detect hot targets on the road, and after fusion with visible light images, they provide a more reliable visual basis for the vehicle's intelligent decision-making.

[0003] However, due to the essential differences in the imaging principles of infrared and visible light images, they have significant differences in grayscale distribution, texture features, and noise characteristics. This makes traditional image registration methods face many challenges when processing these two types of images. Grayscale-based registration methods rely on image grayscale similarity, and the huge difference in grayscale distribution between infrared and visible light images greatly reduces the accuracy and stability of this method. Although feature-based registration methods have certain adaptability, the inconsistency of the features of the two images increases the difficulty of feature extraction and matching. In this context, edge detection technology has unique advantages in infrared and visible light image registration. On the one hand, as an important feature of the target object in the image, the edge contains rich structural information. The edge information extracted by edge detection can outline the contour of the object, and these contours have certain stability and consistency in different modal images. For example, the outline of the object is reflected by grayscale changes in visible light images. Although the grayscale values ​​are different in infrared images, the thermal radiation changes at the boundaries of the object can also form corresponding edges. On the other hand, edge detection can effectively reduce the amount of image data and reduce the complexity of subsequent processing. Compared to processing massive amounts of data for the entire image, edge-based feature extraction and matching can focus more on key information and improve registration efficiency. At the same time, edge detection is relatively insensitive to changes in illumination, which is particularly important for infrared and visible light image registration, because the illumination conditions when the two images are acquired are often quite different. Through edge detection, visible light and infrared heterogeneous images can be converted into homogeneous images, which is conducive to feature extraction and matching.

[0004] At present, the related invention patents are: 1. Application (patent) number: CN202310064850.7, patent name: Image registration method, device, user terminal and medium based on edge features. This patent first performs initial registration on visible light and infrared images, and then extracts the corresponding image edges; then, the gradient values ​​of the extracted edge images are calculated to generate edge features, and the edge features of the two are matched using the correlation coefficient method to determine the registration position. However, the correlation coefficient can only explain the linear correlation between the two. When the extracted edge features are incorrect, the matching will fail. 2. Application (patent) number: CN202410531930.3, patent name: Image data registration method, product, device and medium based on image edge. This patent uses a pre-set initial homography matrix to project the visible light image so that the visible light image is the same size as the infrared image; then, a deep learning method is used to extract the image edges of the visible light and infrared images, and the edge overlap between the two is calculated to obtain the corresponding offset matrix; finally, the initial homography matrix is ​​updated to obtain the final homography matrix to align the visible light and infrared images. This method relies on the accuracy of the initial matrix. Once the initial matrix is ​​set unreasonably, it will affect the subsequent registration effect. 3. Application (patent) number: CN202410704847.1, patent name: A retinal image registration method based on dilated vascular constraints and metric learning. This patent extracts key points from retinal images to align images, using an end-to-end deep learning key point detection and description method. Unlike other deep learning key point detection methods, which obtain variable key points, the SuperRetina method used in this patent obtains a fixed number of key points. After obtaining the key points of the image, the blood vessel segmentation network and the ROP image registration network are used to filter and register the key points.

[0005] It can be seen that the current infrared and visible light image registration technology based on edge detection has the following shortcomings and deficiencies: Feature matching limitations: The feature matching methods used by most existing technologies are relatively simple, such as only considering linear correlation or relying on simple distance metrics, and are insufficient to describe complex feature relationships. Poor adaptability to modal differences: For multimodal images with large differences in imaging principles such as infrared and visible light, the existing technology has not been able to handle their essential differences in grayscale distribution, texture, noise characteristics, etc. It is impossible to effectively overcome the differences between the two modal images in the process of feature extraction, matching and registration, so that the registration accuracy and stability are difficult to guarantee under different lighting, environments and scenes. Computational complexity and error accumulation: Some methods involve complex processing steps, such as multi-stage projection transformation, matrix update, etc., which are prone to error accumulation and affect the final registration results.

[0006] Therefore, how to provide a combined RCF and SuperRetina edge detection feature matching image registration method, system, device and storage medium that can effectively solve the shortcomings and deficiencies in the current edge detection-based infrared and visible light image registration technology is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0007] In view of this, the present invention proposes an image registration method, system, device and storage medium combining RCF and SuperRetina edge detection feature matching.

[0008] In order to achieve the above object, the present invention adopts the following technical solution: A method for image registration combining RCF and SuperRetina edge detection feature matching, comprising: Step 1: Perform initial registration processing on the collected visible light image and infrared image using initial registration parameters; Step 2: Use the edge extraction feature detection model of the joint RCF and SuperRetina to extract the edges of the infrared and visible light images and detect and describe the key points of the features of the bi-optical image after the initial registration process; Step 3: Use the BFMatcher method to match the key points and description points, and use the findHomography method to calculate the affine transformation homography matrix between the matching features based on the points with the best matching correlation. Use the warpPerspective method to perform final registration on the infrared initial registration image after the initial registration processing based on the affine transformation homography matrix.

[0009] Optionally, in step 1, the initial registration parameters are obtained as follows: Using the checkerboard calibration plate as a reference, collect visible light and infrared images at the same time at intervals, find the image pair where the checkerboard calibration plate in the infrared image and the checkerboard calibration plate in the visible light image are completely aligned and overlapped, and record the position coordinates of the current infrared image in the visible light image; The visible light image is cropped according to the position coordinates so that the visible light and infrared images have the same size. The XoFTR method is used to detect and match feature points on the cropped visible light and infrared images, and the initial affine transformation homography matrix is ​​calculated. The position coordinates and the initial affine transformation homography matrix are used as initial registration parameters.

[0010] Optionally, in step 2, RCF in the edge extraction feature detection model is used for edge detection, and SuperRetina is used for feature key point detection and description.

[0011] Optionally, in step 2, SuperRetina includes an encoder and two decoders, the encoder is used to extract a reduced feature map from the image, and the decoder is used to detect and describe feature key points.

[0012] Optionally, step 2 further includes: improving the loss of the edge extraction feature detection model, specifically: Loss to RCF Assign a weight to iteratively update the training model, and the loss of the improved edge extraction feature detection model is as follows:

[0013]

[0014] in, Improved loss for edge extraction feature detection model; for the loss of RCF; for The weight of For the loss of SuperRetina.

[0015] Optionally, in step 3, before using the BFMatcher method to match the key points and the description points, the method further includes: performing post-processing filtering on the key points based on a minimum preset threshold.

[0016] The present invention also provides a combined RCF and SuperRetina edge detection feature matching image registration system using a combined RCF and SuperRetina edge detection feature matching image registration method, comprising: Initial registration module: used to perform initial registration processing on the collected visible light image and infrared image using initial registration parameters; Edge extraction and key point detection and description module: It is used to perform edge extraction and key point detection and description of infrared and visible light images on the bi-optical images after initial registration using the edge extraction feature detection model of the joint RCF and SuperRetina; Final registration module: used to match key points and description points using the BFMatcher method, and based on the points with the best matching correlation, use the findHomography method to calculate the affine transformation homography matrix between matching features, and use the warpPerspective method to perform final registration on the infrared initial registration image after the initial registration processing based on the affine transformation homography matrix.

[0017] The present invention also provides an electronic device, comprising: Memory for storing computer programs; The processor is used to implement the steps of a method for image registration combining RCF and SuperRetina edge detection feature matching when executing a computer program.

[0018] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for image registration combining RCF and SuperRetina edge detection feature matching are implemented.

[0019] It can be seen from the above technical solutions that compared with the prior art, the present invention proposes a method, system, device and storage medium for image registration by combining RCF and SuperRetina edge detection feature matching. First, by manually registering the checkerboard calibration plate and obtaining the initial registration parameters using methods such as XoFTR in the initial stage, the problem of camera field of view difference is effectively solved, providing a stable foundation for subsequent registration. Secondly, in the core feature extraction and matching link, a joint RCF and SuperRetina edge extraction feature detection model is constructed, and the model is further improved by assigning a weight to the loss of RCF, which not only overcomes the image grayscale difference, but also accurately extracts significant feature points, avoiding the limitations of a single feature matching method. Finally, in the final registration process, OpenCV related methods are used for multi-step processing, which significantly improves the accuracy and reliability of registration, reduces the risk of registration failure, and can better cope with infrared and visible light image registration tasks in various complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0021] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0022] Figure 2 Schematic diagram of the network structure of edge extraction feature detection combining RCF and SuperRetina of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Embodiment 1:

[0025] Embodiment 1 of the present invention discloses a method for image registration by combining RCF and SuperRetina edge detection feature matching. Figure 1 As shown, including: Step 1: Use the initial registration parameters to perform initial registration processing on the collected visible light image and infrared image.

[0026] Since the field of view of the visible light camera used is much larger than that of the infrared camera, it is necessary to capture an image with the same scene as the infrared image from the visible light image for initial registration processing.

[0027] The initial registration parameters are obtained as follows: Using the checkerboard calibration plate as a reference, collect visible light and infrared images at the same time at intervals, and manually align them with the help of Photoshop software to find the image pair where the checkerboard calibration plate in the infrared image and the checkerboard calibration plate in the visible light image are completely aligned and overlapped, and record the position coordinates of the current infrared image in the visible light image; The visible light image is cropped according to the position coordinates so that the visible light and infrared images have the same size. The XoFTR method is used to detect and match feature points on the cropped visible light and infrared images, and the initial affine transformation homography matrix is ​​calculated. The position coordinates and the initial affine transformation homography matrix are used as initial registration parameters.

[0028] Step 2: Use the joint RCF and SuperRetina edge extraction feature detection model to perform edge extraction, feature key point detection and description on the infrared and visible light images after the initial registration.

[0029] RCF in the edge extraction feature detection model is used for edge detection. It is improved based on the VGG16 network, makes good use of the rich feature hierarchy, and can be trained through back propagation. SuperRetina is used for feature key point detection and description. It is an end-to-end method with a jointly trainable key point detection and description network. It is aimed at matching retinal images and adopts a semi-supervised training method.

[0030] SuperRetina consists of an encoder and two decoders. The encoder is used to extract reduced feature maps from images, and the decoder is used to detect and describe feature key points.

[0031] To this end, the two methods are combined into a multi-task deep learning model, namely edge extraction and feature key point detection and description tasks.

[0032] It also includes: improving the loss of the edge extraction feature detection model, specifically: Since the entire model first performs edge detection and then detects key feature points, in order to achieve good results, the RCF loss needs to be Assign a weight to iteratively update the training model, and the loss of the improved edge extraction feature detection model is as follows:

[0033]

[0034] in, Improved loss for edge extraction feature detection model; for the loss of RCF; for The weight of For the loss of SuperRetina.

[0035] Combine RCF and SuperRetina edge extraction feature detection network (UnionNet network), such as Figure 2 shown.

[0036] Step 3: Use the BFMatcher method to match the key points and description points, and use the findHomography method to calculate the affine transformation homography matrix between the matching features based on the points with the best matching correlation. Use the warpPerspective method to perform final registration on the infrared initial registration image after the initial registration processing based on the affine transformation homography matrix.

[0037] Before using the BFMatcher method to match key points and description points, it also includes: post-processing and filtering the key points based on a minimum preset threshold, filtering out key points smaller than the threshold to reduce the impact of false detection features.

[0038] Embodiment 2:

[0039] Embodiment 2 of the present invention discloses a combined RCF and SuperRetina edge detection feature matching image registration system using a combined RCF and SuperRetina edge detection feature matching image registration method, comprising: Initial registration module: used to perform initial registration processing on the collected visible light image and infrared image using initial registration parameters; Edge extraction and key point detection and description module: It is used to perform edge extraction and key point detection and description of infrared and visible light images on the bi-optical images after initial registration using the edge extraction feature detection model of the joint RCF and SuperRetina; Final registration module: used to match key points and description points using the BFMatcher method, and based on the points with the best matching correlation, use the findHomography method to calculate the affine transformation homography matrix between matching features, and use the warpPerspective method to perform final registration on the infrared initial registration image after the initial registration processing based on the affine transformation homography matrix.

[0040] Embodiment 3:

[0041] Embodiment 3 of the present invention discloses an electronic device, including: Memory for storing computer programs; The processor is used to implement the steps of a method for image registration combining RCF and SuperRetina edge detection feature matching when executing a computer program.

[0042] Embodiment 4:

[0043] Embodiment 4 of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for image registration combining RCF and SuperRetina edge detection feature matching are implemented.

[0044] The embodiments of the present invention disclose a method, system, device and storage medium for image registration by combining RCF and SuperRetina edge detection feature matching. First, by manually aligning a checkerboard calibration plate and obtaining initial alignment parameters using methods such as XoFTR in the initial stage, the problem of camera field of view difference is effectively solved, providing a stable foundation for subsequent alignment. Secondly, in the core feature extraction and matching link, a joint RCF and SuperRetina edge extraction feature detection model is constructed, and the model is further improved by assigning a weight to the loss of RCF, which not only overcomes the image grayscale difference, but also accurately extracts significant feature points, avoiding the limitations of a single feature matching method. Finally, in the final alignment process, OpenCV related methods are used for multi-step processing, which significantly improves the accuracy and reliability of the alignment, reduces the risk of alignment failure, and can better cope with infrared and visible light image alignment tasks in various complex scenes.

[0045] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0046] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for image registration combining RCF and SuperRetina edge detection feature matching, characterized in that: include: Step 1: Perform initial registration processing on the collected visible light image and infrared image using initial registration parameters; Step 2: Use the edge extraction feature detection model of the joint RCF and SuperRetina to extract the edges of the infrared and visible light images and detect and describe the key points of the features of the bi-optical image after the initial registration process; Step 3: Use the BFMatcher method to match the key points and description points, and use the findHomography method to calculate the affine transformation homography matrix between the matching features based on the points with the best matching correlation. Use the warpPerspective method to perform final alignment on the infrared initial alignment image after the initial alignment processing based on the affine transformation homography matrix.

2. The image registration method combining RCF and SuperRetina edge detection feature matching according to claim 1, characterized in that: In step 1, the initial registration parameters are obtained as follows: Using the checkerboard calibration plate as a reference, collect visible light and infrared images at the same time at intervals, find the image pair where the checkerboard calibration plate in the infrared image and the checkerboard calibration plate in the visible light image are completely aligned and overlapped, and record the position coordinates of the current infrared image in the visible light image; The visible light image is cropped according to the position coordinates so that the visible light and infrared images have the same size. The XoFTR method is used to detect and match feature points on the cropped visible light and infrared images, and the initial affine transformation homography matrix is ​​calculated. The position coordinates and the initial affine transformation homography matrix are used as initial registration parameters.

3. The image registration method combining RCF and SuperRetina edge detection feature matching according to claim 1, characterized in that: In step 2, RCF in the edge extraction feature detection model is used for edge detection, and SuperRetina is used for feature key point detection and description.

4. The image registration method combining RCF and SuperRetina edge detection feature matching according to claim 1, characterized in that: In step 2, the SuperRetina includes an encoder and two decoders, the encoder is used to extract a reduced feature map from the image, and the decoder is used for feature key point detection and description.

5. The image registration method combining RCF and SuperRetina edge detection feature matching according to claim 1, characterized in that: Step 2 also includes: improving the loss of the edge extraction feature detection model, specifically: Loss to RCF Assign a weight to iteratively update the training model, and the loss of the improved edge extraction feature detection model is as follows: ： ： in, The improved loss of the edge extraction feature detection model; for the loss of RCF; for The weight of For the loss of SuperRetina.

6. The image registration method combining RCF and SuperRetina edge detection feature matching according to claim 1, characterized in that: In step 3, before using the BFMatcher method to match the key points and the description points, it also includes: post-processing and filtering the key points based on a minimum preset threshold.

7. A combined RCF and SuperRetina edge detection feature matching image registration system using a combined RCF and SuperRetina edge detection feature matching image registration method according to any one of claims 1 to 6, characterized in that: include: Initial registration module: used to perform initial registration processing on the collected visible light image and infrared image using initial registration parameters; Edge extraction and key point detection and description module: It is used to perform edge extraction and key point detection and description of infrared and visible light images on the bi-optical images after initial registration using the edge extraction feature detection model of the joint RCF and SuperRetina; Final registration module: used to match key points and description points using the BFMatcher method, and based on the points with the best matching correlation, use the findHomography method to calculate the affine transformation homography matrix between matching features, and use the warpPerspective method to perform final registration on the infrared initial registration image after the initial registration processing according to the affine transformation homography matrix.

8. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of a method for image registration combining RCF and SuperRetina edge detection feature matching as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the image registration method combining RCF and SuperRetina edge detection feature matching are implemented as described in any one of claims 1 to 6.

Images