A guided registration method, system, device and storage medium for RGB and infrared image fusion

Abstract

The application provides a guided registration method, system, device and storage medium for RGB and infrared image fusion, and belongs to the technical field of image enhancement based on computer vision. First, the input of the visible light image and the infrared image is obtained, the main target object in the image is identified and detected, and a preliminary target detection frame is generated. Then, taking the target detection frame as spatial prior information, the RGB image segmentation result and the infrared image segmentation result are generated. The segmented two image results are matched in features through a newly designed cross-modal feature matching engine strategy. First, coarse-grained and fine-grained features are extracted through two parallel encoders, then a homography matrix is obtained through a global matcher, distortion refinement and matching sampling, and then the original infrared image is subjected to homographic transformation to obtain a final registration image. The application is superior to the prior art in registration accuracy and stability, and provides a robust solution for the registration of infrared and visible light images.

Description

Technical Field

[0001] This invention belongs to the field of image enhancement technology based on computer vision, and particularly relates to a guided registration method, system, device and storage medium for RGB and infrared image fusion. Background Technology

[0002] In harsh lighting conditions such as dense fog and low illumination, infrared cameras can capture the thermal radiation of objects to create images, thus effectively detecting targets. However, the quality of infrared images is inferior to that of visible light images, which contain richer scene information but are easily affected by lighting conditions. Infrared and visible light fusion technology combines the advantages of both, fusing the information from two images into a single image. This allows for the extraction of both infrared and visible light information from a single image, promoting the development of multiple tasks in computer vision. However, without precise registration of the infrared and visible light images, the fusion effect may be unsatisfactory. Therefore, finding a technology that can accurately register infrared and visible light images is crucial.

[0003] Region-based and feature-based methods are two existing approaches to image registration. Region-based registration algorithms primarily rely on a reference image, using various similarity metrics to find the correlation between the two images in the reference image. This method has been widely used in registering multimodal images with intensity differences. However, these methods may lead to local extrema and incur significant computational costs when dealing with infrared and visible light images with large modal differences. Feature-based methods, on the other hand, iteratively search for the optimal similarity between two images, performing pixel-level expansion of image feature information to obtain corner points, edge segments, feature regions, etc., and then using a matching algorithm to match the feature points of the two images.

[0004] Regarding the problem of feature matching, many scholars have conducted research on extracting the same or similar features from two different images. Among them, the classic methods are Scale Invariant Feature Transform (SIFT) and the faster but less accurate Speed-Up Robust Feature Transform (SURF). However, feature matching first extracts the stronger or more distinct features from the image. For images with indistinct intensity differences, very few or even almost no feature points are extracted. This results in very few matching point pairs, and even incorrect matching may occur, so the final matching effect is not ideal. Summary of the Invention

[0005] To address the above problems, the first aspect of the present invention provides a guided registration method for RGB and infrared image fusion, comprising the following steps:

[0006] Step 1: Receive and acquire the RGB visible light image r and the corresponding infrared image t in real time;

[0007] Step 2: Based on the input of the visible light image r and the infrared image t, identify and detect the main target objects in the images, and generate preliminary target detection boxes;

[0008] Step 3: Using the target detection bounding box as spatial prior information, the target bounding box is transformed into spatial attention weights through a cue coding mechanism, guiding the focus to a specific region for pixel-level segmentation, generating RGB image segmentation result sr and infrared image segmentation result st;

[0009] Step 4: Perform feature matching on the segmented sr and st. First, extract coarse-grained and fine-grained features using two parallel encoders. The coarse-grained feature encoder captures the global structural information of the image, while the fine-grained feature encoder focuses on local texture details. The coarse-grained features are processed by a global matcher to obtain coarse-grained mappings and confidence scores from the overall scene structure. The coarse-grained mappings and confidence scores obtained from the fine-grained features and the global matcher are further filtered through distortion and thinning to obtain dense mappings and their confidence scores. Then, a set of reliable matches is selected through matching sampling to generate a homography matrix. Finally, the obtained homography matrix is ​​homography transformed with the original input image t to obtain the final registered image ft.

[0010] Preferably, step 2 employs a multi-scale feature extraction network. Starting from the input image, features are extracted step by step through a series of convolutional layers and attention modules, and the final target coordinate information is output through three detection heads. The backbone of the network is composed of multiple convolutional layers and C3k2 modules stacked alternately to form a deep feature extraction network. The feature pyramid structure achieves the fusion of features at different scales through upsampling and feature connection operations. Finally, the feature maps at different scales are processed by the three detection heads respectively to output the target localization result, i.e., the target detection box.

[0011] Preferably, the multi-scale feature extraction network specifically comprises:

[0012] The system receives input images r and t, extracts preliminary features through two convolutional layers, and then progressively downsamples the feature maps to different scales through a backbone network to extract multi-level features. The backbone network consists of four "Conv+C3k2" modules. The network ends include an SPPF module, which expands the receptive field through multi-scale pooling, and a C2PSA module, which applies an attention mechanism to optimize deep features. In the feature pyramid, deep features are first upsampled and fused with mid-level features, and the fused features are then upsampled again and fused with shallow features. After each fusion, the C3k2 module is used to enhance the feature representation. The system also includes three parallel detection heads, each processing features at different scales: Head1 handles deep features for detecting large targets, Head2 handles mid-level features for detecting medium-sized targets, and Head3 handles shallow features for detecting small targets. Finally, the Coordinates module integrates the prediction results of the three detection heads, applies the NMS algorithm to eliminate duplicate detections, and outputs the target's location coordinates, size, and category information.

[0013] Preferably, step 3 employs a semantic segmentation enhancement module. Based on the obtained target detection box, rich visual features are extracted by an image encoder, and then the user-provided interactive information is processed by a mask decoder and a cue decoder to finally obtain an accurate segmentation mask; specifically including:

[0014] A multi-layer image encoder structure extracts rich visual features to generate multi-scale image embeddings. These image embeddings are dense feature maps containing both local and global semantic information of the image. Simultaneously, user interaction prompts are received from the left side. These prompts are first processed by a prompt decoder and converted into standardized prompt embeddings. The prompt embeddings are then fed into a mask decoder, which associates the prompt information with image features through an attention mechanism to generate preliminary mask features. These mask features, along with the mask input from the top, are processed through a convolutional layer (conv) and then fused with the image embeddings to produce an enhanced mask representation. Based on the fused features, the mask decoder generates the final segmentation mask through multi-head attention and a feedforward network, identifying the image region corresponding to the user prompt.

[0015] Preferably, the global matcher in step 4 is specifically:

[0016] Based on the coarse-grained features obtained from the encoder, key points are detected in two images and descriptors are generated. By calculating the similarity matrix between the descriptors of the two images, the point with the highest similarity in one image to the other image is obtained. Finally, the coarse-grained mapping and confidence are obtained.

[0017] Preferably, the distortion thinning in step 4 is based on the fine-grained features obtained by the encoder and the coarse-grained mapping and confidence obtained by the global matcher to estimate a global geometric transformation. Then, for each matching point in one image, the position of the corresponding point is fine-tuned in another image. The global geometric transformation and matching position are iteratively transformed until the dense mapping and its confidence are converged.

[0018] Preferably, the matching sampling in step 4 is based on the dense mapping and confidence obtained by distortion thinning to eliminate matches with low confidence. The image is divided into grids, and one match is randomly retained in each grid to avoid local clustering. Finally, the final matching point pairs are output, and the homography matrix is ​​calculated from the final matching point pairs.

[0019] A second aspect of the present invention provides a guided registration system for RGB and infrared image fusion, comprising a receiving module, a multi-scale feature extraction network module, a semantic segmentation enhancement module, and a cross-modal feature matching engine network module;

[0020] The receiving module is used to receive and acquire the RGB visible light image r and the corresponding infrared image t;

[0021] The multi-scale feature extraction network module, based on the input of visible light image r and infrared image t, identifies and detects the main target objects in the image and generates a preliminary target detection box;

[0022] The semantic segmentation enhancement module uses the target detection box as spatial prior information. Through a cue coding mechanism, the target box is transformed into spatial attention weights, which guide the focus on a specific region for pixel-level segmentation, generating RGB image segmentation result sr and infrared image segmentation result st.

[0023] The cross-modal feature matching engine network module performs feature matching on the segmented sr and st. First, coarse-grained and fine-grained features are extracted by two parallel encoders. The coarse-grained feature encoder captures the global structural information of the image, while the fine-grained feature encoder focuses on local texture details. The coarse-grained features are processed by a global matcher to obtain coarse-grained mappings and confidence scores from the overall scene structure. The coarse-grained mappings and confidence scores obtained by the fine-grained features and the global matcher are further filtered through distortion and thinning to obtain dense mappings and their confidence scores. Then, a set of reliable matches is selected through matching sampling to generate a homography matrix. Finally, the obtained homography matrix is ​​subjected to homography transformation with the original input image t to obtain the final registered image ft.

[0024] A third aspect of the present invention also provides a guided registration device for RGB and infrared image fusion, the device comprising at least one processor and at least one memory coupled together; the memory storing a computer-executable program; when the processor executes the computer-executable program stored in the memory, the processor performs a guided registration method for RGB and infrared image fusion as described in the first aspect.

[0025] A fourth aspect of the present invention also provides a computer-readable storage medium storing a computer-executable program, which, when executed by a processor, causes the processor to perform a guided registration method for RGB and infrared image fusion as described in the first aspect.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] This invention effectively removes or dilutes background information unrelated to the target through a semantic segmentation enhancement module, significantly enhancing the shape features of the target to be registered. This process not only improves the efficiency of feature extraction but also significantly increases the number of extractable feature points, making image matching cleaner and more accurate.

[0028] This invention proposes a novel cross-modal feature matching engine strategy that extracts coarse-grained and fine-grained features through two parallel encoders. The coarse-grained feature encoder captures the global structural information of the image, while the fine-grained feature encoder focuses on local texture details, comprehensively extracting image features. The two types of features are processed by a global matcher, distortion thinning, and matching sampling to obtain a homography matrix. Based on the input original infrared image and the homography matrix, the final registered image is obtained through homography transformation. This approach outperforms existing methods in terms of registration accuracy and stability, providing a robust solution for the registration of infrared and visible light images.

[0029] This invention combines a multi-scale feature extraction network, a semantic segmentation enhancement module, and a cross-modal feature matching engine network to achieve accurate spatial alignment of images. Through the collaborative work of multiple modules, this method can effectively reduce background interference and improve the accuracy and efficiency of image registration, making it suitable for image alignment tasks in complex scenarios. Attached Figure Description

[0030] Figure 1 This is the overall logic diagram of the fusion-guided registration method of the present invention.

[0031] Figure 2 This is a diagram of the multi-scale feature extraction network structure of the present invention.

[0032] Figure 3 This is a structural diagram of the semantic segmentation enhancement module of the present invention.

[0033] Figure 4 This is a visualization of the experimental results in an embodiment of the present invention.

[0034] Figure 5 This is a simplified structural diagram of the guidance and registration device of the present invention. Detailed Implementation

[0035] This invention mainly proposes a guided registration method and system structure for RGB and infrared image fusion, the specific implementation logic of which is as follows: Figure 1 As shown. It includes the following processes:

[0036] Step 1: Receive and acquire the RGB visible light image r and the corresponding infrared image t in real time;

[0037] Step 2: Based on the input of the visible light image r and the infrared image t, identify and detect the main target objects in the images, and generate preliminary target detection boxes;

[0038] Step 3: Using the target detection bounding box as spatial prior information, the target bounding box is transformed into spatial attention weights through a cue coding mechanism, guiding the focus to a specific region for pixel-level segmentation, generating RGB image segmentation result sr and infrared image segmentation result st;

[0039] Step 4: Perform feature matching on the segmented sr and st. First, extract coarse-grained and fine-grained features using two parallel encoders. The coarse-grained feature encoder captures the global structural information of the image, while the fine-grained feature encoder focuses on local texture details. The coarse-grained features are processed by a global matcher to obtain coarse-grained mappings and confidence scores from the overall scene structure. The coarse-grained mappings and confidence scores obtained from the fine-grained features and the global matcher are further filtered through distortion and thinning to obtain dense mappings and their confidence scores. Then, a set of reliable matches is selected through matching sampling to generate a homography matrix. Finally, the obtained homography matrix is ​​homography transformed with the original input image t to obtain the final registered image ft.

[0040] To achieve the above method, this invention provides a registration system that mainly includes a multi-scale feature extraction network module, a semantic segmentation enhancement module, and a cross-modal feature matching engine network module. Through the collaborative work of multiple modules, it can effectively reduce background interference and improve the accuracy and efficiency of image registration, making it suitable for image alignment tasks in complex scenarios.

[0041] The invention will be further described below with reference to specific embodiments.

[0042] I. Multi-scale Feature Extraction Network Module

[0043] Network module structure as follows Figure 2As shown, this module receives a standard RGB image with a size of 640×640×3. The input image first passes through a series of convolutional layers and downsampling operations to form the backbone network for feature extraction. The backbone network adopts the CSPDarknet structure, consisting of multiple convolutional layers and residual connections. The first convolutional layer uses 32 3×3 filters (stride 2) to convert the input image into a 320×320×32 feature map. Subsequently, the image passes through a second convolutional layer (64 3×3 filters) for further feature extraction. Next, the feature map undergoes a first downsampling operation with a stride of 2 (labeled "stride 2"), reducing the resolution to 160×160×128. The feature map continues to be processed through a third convolutional layer, followed by a second downsampling operation with a stride of 2, reducing the resolution to 80×80×256. Subsequently, the feature map undergoes a fourth convolutional layer, followed by a third downsampling operation with a stride of 2, reducing the resolution to 40×40×512. The feature map is then processed through a fifth convolutional layer, followed by a fourth downsampling operation with a stride of 2, ultimately reducing the resolution to 20×20×1024.

[0044] At the end of the backbone network, the model applies a spatial pyramid pooling module (labeled "spatial pyramid pooling") to expand the receptive field and enhance feature representation through multi-scale pooling operations. Next, a self-attention mechanism is applied to the feature map, further optimizing the feature representation by calculating the relationships between feature channels, outputting an enhanced 20×20×1024 feature map. The Feature Pyramid Network (PANet) then handles multi-scale feature fusion. First, the deepest feature layer (20×20×1024) is upsampled to 40×40 and fused with the corresponding feature layer (40×40×512) in the backbone network through a connection layer (labeled "connection layer"). The fused features are then processed by a convolutional layer with a stride of 2 to generate mid-level features of 40×40×512. These mid-level features are then upsampled again to 80×80 and fused with the corresponding feature layer (80×80×256) in the backbone network through another connection layer. The fused features are processed by a convolutional layer to generate shallow features of 80×80×256.

[0045] The detection head consists of three parallel branches. Detection head 1 processes deep features (20×20×1024), focusing on detecting large targets; detection head 2 processes mid-level features (40×40×512), targeting medium-sized targets; and detection head 3 processes shallow features (80×80×256), responsible for detecting small targets. Each detection head contains multiple convolutional layers and one output layer. Detection head 1 outputs a 20×20×(4+1+C) tensor, where 4 represents the bounding box coordinates (x, y, width, height), 1 represents the target confidence score, and C represents the number of classes. Detection head 2 outputs a 40×40×(4+1+C) tensor, and detection head 3 outputs an 80×80×(4+1+C) tensor.

[0046] Finally, the outputs of the three detectors are integrated and non-maximum suppression (NMS) algorithm (IoU threshold is usually 0.45) is applied to eliminate duplicate detections and generate the final detection result.

[0047] II. Semantic Segmentation Enhancement Module

[0048] Module structure as follows Figure 3As shown, the processing flow begins with the image decoder on the right, which receives the input image (typically a 1024×1024×3 RGB image). The image decoder is based on the Vision Transformer (ViT) architecture and consists of 12 Transformer blocks, each containing a multi-head self-attention mechanism (12 attention heads, each 64-dimensional) and a feedforward neural network. The decoder first divides the input image into 16×16 image patches, generating 64×64 patches, each linearly projected into a 768-dimensional embedding space. These patch embeddings are then enhanced with positional encoding to add spatial location information. The enhanced embedding sequence is processed by the Transformer blocks to capture the local and global contextual relationships of the image, ultimately outputting a dense image embedding with dimensions of 64×64×256, represented as "image embedding" in the figure. Simultaneously, the model receives user interaction prompts from the left, which can be clicks (positive or negative dots), selection boxes, or text descriptions, collectively referred to as the "image to be segmented". These cues are first processed by a prompt decoder, which consists of two Transformer layers that convert different types of cues into standardized cue embeddings of dimension N×256, where N is the number of cue points. The cue embeddings are then fed into a mask decoder, a lightweight Transformer containing two self-attention layers and two cross-attention layers. The mask decoder uses a cross-attention mechanism to associate the cue information with image features, generating preliminary mask features.

[0049] Meanwhile, the mask input from the top (if any) is processed by a convolutional layer (Conv) to generate a mask embedding. The mask embedding is fused with the image embedding and cue embedding through an addition operation (the "⊕" symbol in the diagram) to produce an enhanced mask representation. Based on these fused features, the mask decoder generates mask embeddings with dimensions 256×256×1 using a multi-head attention mechanism and a feedforward network. These embeddings are upsampled (typically by 4x bilinear interpolation) to transform into the final segmentation mask with dimensions 1024×1024×1, represented as the "segmented image" in the diagram. Each pixel value in the mask ranges from 0 to 1, representing the probability that the pixel belongs to the target object.

[0050] III. Cross-modal feature matching engine network module

[0051] Distortion thinning processes features at different scales through two parallel paths. One is a fine-grained feature path, which extracts local texture details through a four-layer convolutional network (64, 128, 256, and 512 channels respectively), outputting a 160×160×512 feature map. These features capture the fine structure of the image, such as edges, corners, and texture variations. The other is a coarse-grained feature path, which processes features with a larger receptive field through a global matcher. The global matcher first enhances the feature representation through a self-attention mechanism (8 attention heads, 64 dimensions each), and then computes phase correlation and confidence scores across the image. The phase correlation is represented as an 80×80×2 tensor, describing pixel-level correspondences; the confidence score is an 80×80×1 tensor used to identify reliable matching regions.

[0052] There is a bidirectional information flow between the two feature paths: fine-grained features guide the optimization of coarse-grained features, while the global consistency constraints of coarse-grained features, in turn, improve the matching accuracy of fine-grained features. This complementary mechanism is achieved through a feature fusion layer that integrates information from both scales using 1×1 convolutions and residual connections. Two encoders process the optimized fine-grained and coarse-grained features respectively. Each encoder contains three residual blocks, each consisting of two 3×3 convolutional layers and a skip connection. The encoders output phase-level features, which encode pixel-level correspondences and deformation fields.

[0053] The global matcher process obtains a segmentation mask based on the semantic segmentation enhancement module and calculates dense features (256-dimensional vectors per pixel) using a local feature extractor. A cross-correlation algorithm is used to calculate the similarity matrix between feature points, and a soft-maximum function is used to generate preliminary matching relationships. Simultaneously, this module also calculates a confidence score (a scalar value between 0 and 1) for each matching point, used for subsequent filtering of unreliable matches. These estimated mappings and confidence scores are encoded as a 640×640×3 tensor, where the first two channels represent the mapping coordinates and the third channel represents the confidence score.

[0054] Matching sampling is based on dense mapping and confidence levels, and estimates global geometric transformation parameters using a weighted RANSAC algorithm. This module first selects the top N most reliable matching points (typically N=2000) based on confidence levels, and then finds the optimal transformation model through iterative sampling (typically 5000 iterations). The calculated homography matrix is ​​a 3×3 transformation matrix that describes the projection relationship from the source image to the target image.

[0055] The Warp module uses a homography matrix to perform homography transformation on the source image to obtain the final registered image.

[0056] IV. Explanation of Experimental Results

[0057] Tables 1 and 2 show the results obtained from testing the present invention on two different datasets. The experimental results are visualized as follows: Figure 4 As shown in the experimental results, the proposed method consistently outperforms other existing methods in all aspects. Specifically, the method exhibits more accurate performance in registering infrared and visible light images. Notably, our method maintains high accuracy even at different shooting heights and angles, ensuring accurate image registration. This advantage enables our technology to be effectively applied in complex and varied environments, further demonstrating its superiority and reliability in the field of image processing.

[0058] Table 1. Registration results on public datasets

[0059]

[0060] Table 2. Registration results on the private dataset

[0061]

[0062] like Figure 5 As shown, this invention also provides a guided registration device for RGB and infrared image fusion. The device includes at least one processor and at least one memory, as well as a communication interface and an internal bus. The memory stores a computer-executable program. When the processor executes the computer-executable program stored in the memory, it can execute a guided registration method for RGB and infrared image fusion. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus in the accompanying drawings is not limited to only one bus or one type of bus. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk storage device, or a USB flash drive, external hard drive, read-only memory, disk, or optical disk, etc.

[0063] The device may be provided as a terminal, server, or other type of device. In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0064] The present invention also provides a computer-readable storage medium storing a computer-executable program, which, when executed by a processor, enables the processor to perform a guided registration method for RGB and infrared image fusion.

[0065] Specifically, a system, apparatus, or device may be provided equipped with a readable storage medium on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system, apparatus, or device reads and executes the instructions stored in the readable storage medium. In this case, the program code read from the readable medium itself can implement the functions of any of the embodiments described above, therefore, the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0066] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0067] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A guided registration method for RGB and infrared image fusion, characterized in that, The method comprises the following steps: Step 1, real-time receiving an RGB visible light image r and a corresponding infrared image t; Step 2, based on the input of the visible light image r and the infrared image t, identifying and detecting the main target object in the image to generate a preliminary target detection frame; Step 3, taking the target detection frame as spatial prior information, converting the target frame into spatial attention weight through a prompt coding mechanism, guiding focusing on a specific area for pixel-level segmentation to generate an RGB image segmentation result sr and an infrared image segmentation result st; Step 4, performing feature matching on the segmented sr and st, first extracting coarse-grained and fine-grained features through two parallel encoders, the coarse-grained feature encoder captures the global structural information of the image, while the fine-grained feature encoder focuses on local texture details, the coarse-grained features pass through a global matcher to obtain coarse-grained mapping and confidence from the overall scene structure, the fine-grained features and the coarse-grained mapping and confidence obtained by the global matcher are further screened through distortion refinement to obtain dense mapping and its confidence, then a group of reliable matches are selected through matching sampling to generate a homography matrix, and finally the obtained homography matrix is subjected to homography transformation with the original input image t to obtain a final registration image ft; The global matcher specifically detects key points in the two images based on the coarse-grained features obtained by the encoder and generates a descriptor, the similarity matrix between the two image descriptors is calculated to obtain the point with the highest similarity of each feature point in one image in the other image, and finally the coarse-grained mapping and confidence are obtained; The distortion refinement is based on the fine-grained features obtained by the encoder and the coarse-grained mapping and confidence obtained by the global matcher to estimate a global geometric transformation, then for each matching point in one image, the position of the corresponding point in the other image is fine-tuned, and the global geometric transformation and the matching position are iteratively transformed until convergence to obtain the dense mapping and its confidence; The matching sampling is based on the dense mapping and confidence obtained by the distortion refinement to eliminate matches with low confidence, the image is divided into grids, one match in each grid is randomly retained to avoid local aggregation, and finally the final matching point pair is output, and the homography matrix is calculated through the final matching point pair.

2. The guided registration method for RGB and infrared image fusion according to claim 1, characterized in that: Step 2 adopts a multi-scale feature extraction network, which gradually extracts features from the input image through a series of convolutional layers and attention modules, and outputs the final target coordinate information through three detection heads; the main part of the network is composed of a plurality of convolutional layers and C3k2 modules stacked alternately to form a deep feature extraction network; And the feature pyramid structure realizes the fusion of features of different scales through upsampling and feature connection operations, and finally processes feature maps of different scales through three detection heads respectively to output target positioning results, i.e. target detection frames.

3. A guided registration method for RGB and infrared image fusion as claimed in claim 2, wherein, The multi-scale feature extraction network specifically comprises: The input image r and t are received, and preliminary features are extracted through two convolutional layers; then, through a backbone network, the feature map is gradually down-sampled to different scales to extract multi-level features, and the backbone network is composed of four groups of "Conv+C3k2" modules; the network end includes an SPPF module, which expands the receptive field through multi-scale pooling, and a C2PSA module, which applies an attention mechanism to optimize deep features; in the feature pyramid part, the deep features are first up-sampled and fused with the middle-level features, the fused features are again up-sampled and fused with the shallow features, and the feature expression is enhanced through the C3k2 module after each fusion; three parallel detection heads are also included, which process features of different scales: Head1 is responsible for deep features and is used for detecting large targets, Head2 processes middle-level features and is aimed at medium targets, and Head3 processes shallow features and focuses on small target detection; finally, the prediction results of the three detection heads are integrated through the Coordinates module, the NMS algorithm is applied to eliminate repeated detection, and the position coordinates, size and category information of the target are output.

4. A guided registration method for RGB and infrared image fusion as claimed in claim 1, characterized in that: The step 3 adopts a semantic segmentation enhancement module, based on the obtained target detection frame, extracts rich visual features through an image encoder, then processes the user-provided interaction information through a mask decoder and a prompt decoder, and finally obtains an accurate segmentation mask; Specifically comprising: Rich visual features are extracted through a multi-layer image encoder structure to generate multi-scale image embeddings, which are dense feature maps containing local and global semantic information of the image; at the same time, user interaction prompts are received from the left, and the prompts are first processed by a prompt decoder to convert them into standardized prompt embeddings; the prompt embeddings are then sent to a mask decoder, which associates the prompt information with the image features through an attention mechanism to generate preliminary mask features; the mask features and the mask input from the top are processed by a convolution layer conv, and then fused with the image embeddings to produce enhanced mask representations; based on the fused features, the mask decoder generates the final segmentation mask through a multi-head attention and a feedforward network, identifying the image region corresponding to the user prompt.

5. A guided registration system for RGB and infrared image fusion, characterized by: It comprises a receiving module, a multi-scale feature extraction network module, a semantic segmentation enhancement module, and a cross-modal feature matching engine network module. The receiving module is used to receive and obtain an RGB visible light image r and a corresponding infrared image t; The multi-scale feature extraction network module identifies and detects the main target objects in the image based on the input of the visible light image r and the infrared image t, and generates preliminary target detection frames; The semantic segmentation enhancement module uses the target detection frame as spatial prior information, converts the target frame into spatial attention weights through a prompt encoding mechanism, guides focusing on a specific area for pixel-level segmentation, and generates an RGB image segmentation result sr and an infrared image segmentation result st; The cross-modal feature matching engine network module performs feature matching on the segmented sr and st. First, coarse-grained and fine-grained features are extracted through two parallel encoders. The coarse-grained feature encoder captures global structural information of the image, and the fine-grained feature encoder focuses on local texture details. The coarse-grained features are subjected to a global matcher to obtain coarse-grained mapping and confidence from the overall scene structure. The fine-grained features and the coarse-grained mapping and confidence obtained by the global matcher are subjected to distortion refinement to further screen dense mapping and its confidence. Then, a set of reliable matches is selected through matching sampling to generate a homography matrix. Finally, the obtained homography matrix is subjected to homographic transformation with the original input image t to obtain a final registration image ft. The global matcher specifically detects key points in the two images and generates descriptors based on the coarse-grained features obtained by the encoder. By calculating the similarity matrix between the descriptors of the two images, the most similar point of each feature point in one image in the other image is obtained, and finally the coarse-grained mapping and confidence are obtained. The distortion refinement estimates a global geometric transformation based on the fine-grained features obtained by the encoder and the coarse-grained mapping and confidence obtained by the global matcher. Then, for each matching point in one image, the position of the corresponding point in the other image is fine-tuned. The global geometric transformation and the matching position are iteratively transformed until convergence to obtain dense mapping and its confidence. The matching sampling eliminates low-confidence matches based on the dense mapping and confidence obtained by the distortion refinement. The image is divided into grids, and one matching in each grid is randomly retained to avoid local aggregation. Finally, the final matching point pair is output, and the homography matrix is calculated based on the final matching point pair.

6. A guided registration device for RGB and infrared image fusion, characterized by: The device includes at least one processor and at least one memory, which are coupled; the memory stores a computer execution program; when the processor executes the computer execution program stored in the memory, the processor executes the guided registration method for RGB and infrared image fusion according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer execution program, and when the processor executes the computer execution program, the processor executes the guided registration method for RGB and infrared image fusion according to any one of claims 1 to 4.

Images