A visible-infrared dual-modal image registration method based on deep learning

Abstract

The present invention discloses a method for visible-light-infrared dual-modal image registration based on deep learning, which includes: acquiring a visible light image and an infrared image of the same scene; image preprocessing; performing feature extraction on the preprocessed visible light image and the preprocessed infrared image respectively, and using a dual-attention mechanism to enhance the feature extraction results; performing multi-scale feature fusion on the enhanced visible light image features and the enhanced infrared image features to generate cross-modal fusion features; inputting the cross-modal fusion features into a deformation field estimation network to generate an estimation result of the deformation field; and applying the deformation field estimation result to the infrared image to align the corresponding visible light image. The method of the present invention effectively solves the cross-modal registration problem between visible light images and infrared thermal imaging for non-contact prediction of pig body temperature, achieves high-precision pixel-level alignment, and provides a technical foundation for non-contact pig body temperature monitoring.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to a visible light-infrared dual-modal image registration method based on deep learning. Background Art

[0002] In large-scale pig farms, pig health monitoring is a key factor affecting pig growth and production capacity. Traditional body temperature detection methods mainly rely on contact methods such as rectal temperature measurement, but this method has many significant drawbacks. First, this method requires staff to manually capture and fix pigs, which not only significantly increases labor costs, but also inevitably causes stress reactions in the pigs during the operation, which in turn has a negative impact on their growth status and health. Secondly, this method cannot achieve large-scale continuous monitoring, making it difficult to detect feverish pigs in a timely manner, delaying the best time for disease warning. More seriously, during epidemics, contact temperature measurement may lead to the risk of cross-infection, which in turn exacerbates the spread of the epidemic.

[0003] In response to the above problems, infrared thermal imaging technology has been introduced into the field of pig body temperature monitoring in recent years, attempting to achieve non-contact temperature measurement. However, relying solely on infrared thermal imaging technology still faces insurmountable technical bottlenecks. The resolution of infrared images is generally low (usually only 320×240 pixels), and it is impossible to accurately identify individual pigs and specific parts of their body surface, resulting in limited temperature measurement accuracy. At the same time, thermal imaging systems have difficulty distinguishing between different individual pigs, making individual tracking and historical data comparison almost impossible. In addition, environmental factors such as air humidity and airflow have a significant impact on infrared imaging results, further reducing the reliability of the measurement. Most importantly, infrared images lack the rich appearance feature information provided by visible light, and cannot obtain the complete visual characteristics of pigs.

[0004] While visible light imaging systems can provide high-resolution visual information of pigs, they cannot directly measure body temperature. An ideal solution would combine the advantages of both modalities, but currently, there is a lack of systems on the market that can effectively register visible light and infrared images, making it impossible to accurately monitor and track individual pig temperatures. Traditional image registration methods face significant challenges in this application scenario, primarily due to two core technical difficulties. First, consistent cross-modal feature representation is required. Visible light and infrared images have completely different imaging mechanisms: the former reflects the reflective properties of an object's surface, while the latter reflects the temperature distribution on the surface. This fundamental difference makes it difficult to establish a direct correspondence between the two modalities using traditional intensity- or gradient-based features. Second, there is the issue of compensating for the non-rigid deformation of the pig's body. Pigs undergo various posture changes and body flexions during movement, requiring a complex non-rigid deformation model to accurately describe and compensate for these changes. Furthermore, the significant resolution difference between the two modalities and environmental factors (such as lighting and shadow variations) further complicate and instability registration. Summary of the Invention

[0005] In view of this, the object of the present invention is to provide a visible light-infrared dual-modal image registration method based on deep learning.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A visible light-infrared dual-modal image registration method based on deep learning, comprising:

[0008] Step 1: Acquire two images of the same scene in different modes, namely, a visible light image and an infrared image;

[0009] Step 2: Preprocess the visible light image and infrared image separately;

[0010] Step 3: Feature extraction is performed on the preprocessed visible light image and the preprocessed infrared image respectively, and the feature extraction results are enhanced using the dual attention mechanism to generate enhanced visible light image feature maps and enhanced infrared image feature maps. The dual attention mechanism includes channel attention and spatial attention.

[0011] Step 4: Perform multi-scale feature fusion on the enhanced visible light image features and the enhanced infrared image features to generate cross-modal fusion features;

[0012] Step 5: Input the cross-modal fusion features into the deformation field estimation network to generate an estimation result of the deformation field. The deformation field estimation network is a layered deformation field estimation network based on deformable convolution. The deformation field estimation network contains multiple layers of deformation field estimation subnetworks, and each layer of deformation field estimation subnetwork generates the deformation field estimation result corresponding to that layer.

[0013] Step 6: Apply the estimated deformation field to the infrared image to register the corresponding visible light image.

[0014] Furthermore, the preprocessing of visible light images includes denoising the visible light images using a Gaussian filter and enhancing the image contrast through histogram equalization; the preprocessing of infrared images includes removing salt and pepper noise in the infrared images using a median filter and enhancing the local contrast in the infrared images using a contrast-limited adaptive histogram equalization method. The preprocessing also includes preliminary geometric alignment of the enhanced visible light image and the enhanced infrared image through SIFT feature point detection and matching method.

[0015] Furthermore, step 3 specifically includes:

[0016] S31: Use the pre-trained ResNet-50 network to perform initial feature extraction on the pre-processed visible light image and the pre-processed infrared image, respectively, to generate a visible light image feature map and an infrared image feature map;

[0017] S32: Use a channel attention mechanism to enhance the visible light image feature map and the infrared image feature map respectively to generate a first visible light image enhanced feature map and a first infrared image enhanced feature map;

[0018] S33: Use the spatial attention mechanism to enhance the first visible light image enhancement feature map and the first infrared image enhancement feature map respectively to generate a second visible light image enhancement feature map and a second infrared image enhancement feature map, i.e., an enhanced visible light image feature map and an enhanced infrared image feature map.

[0019] Furthermore, step 4 specifically includes:

[0020] S41: Constructing multi-scale feature pyramids for the enhanced visible light image feature map and the enhanced infrared image feature map respectively;

[0021] S42: performing channel unification processing on the visible light image features of each scale and the infrared image features of each scale, respectively, so that the visible light image features of different scales and the infrared image features of different scales have the same channel dimension;

[0022] S43: After the channels are unified, the visible light image features of different scales and the infrared image features of different scales are spatially aligned using bilinear interpolation, so that all features are aligned to a unified spatial size;

[0023] S44: For each scale, respectively, the spatially aligned visible light image features and infrared image features of the scale are used to calculate the importance weights of the visible light image features and the importance weights of the infrared image features at the scale through global average pooling and multi-layer perceptron;

[0024] S45: normalizing the importance weights of visible light image features and infrared image features at each scale respectively;

[0025] S46: Using the normalized importance weights of each scale, weighted fusion is performed on the spatially aligned visible light image features and the spatially aligned infrared image features of different scales to obtain fused features corresponding to the visible light image and fused features corresponding to the infrared image;

[0026] S47: Perform cross-modal feature fusion on the fusion features corresponding to the visible light image and the fusion features corresponding to the infrared image through learnable modal weight parameters to obtain cross-modal fusion features.

[0027] Furthermore, step 5 includes:

[0028] S51: Construct the cross-modal fusion features into a multi-layer feature pyramid structure. The spatial resolution of each layer decreases in sequence. The layer with the highest resolution is the first layer, and the layer with the lowest resolution is the top layer. Each layer of features corresponds to a deformation field estimation subnetwork.

[0029] S52: For the top layer, use the layer features to perform deformation field estimation to generate the deformation field estimation result of the top layer;

[0030] S53: For each layer except the top layer, the features of the layer and the up-sampled result of the deformation field estimation result of the previous layer are used as the input of the deformation field estimation sub-network of the layer, thereby obtaining the deformation field estimation result of the layer;

[0031] S54: Taking the deformation field estimation result of the first layer as the total deformation field estimation result D.

[0032] Furthermore, the output of the deformation field estimation network also includes a branch for predicting logarithmic variance. The method also includes performing Bayesian uncertainty modeling at the highest resolution layer of the deformation field estimation network. Bayesian uncertainty includes epistemic uncertainty and aleatoric uncertainty. The Bayesian uncertainty modeling includes:

[0033] The weight parameters in the first-layer deformation field estimation subnetwork are modeled as probability distributions in a Bayesian framework, and the Dropout layer activation is kept during the inference phase.

[0034] By performing multiple forward propagations and sampling the predicted results of the deformation field, the mean and variance of the deformation field, i.e., the mean and variance of the epistemic uncertainty, are calculated;

[0035] The cross-modal fusion features are passed through the branch of predicting the logarithmic variance to generate the variance of accidental uncertainty;

[0036] The total uncertainty variance is obtained by combining the epistemic uncertainty variance and the aleatoric uncertainty variance, and the total uncertainty variance is used to generate a confidence map to evaluate the registration quality.

[0037] Furthermore, the method further includes, after generating an estimation result of the total deformation field using the deformation field estimation network, correcting the deformation field using a graph convolutional network, specifically including:

[0038] Construct a graph structure, that is, the total deformation field estimation result D∈R H×W×2 Each pixel position is defined as a node in the graph, and edges are established between nodes that meet the Manhattan distance constraint. The node set is V = {v i,j |i∈[1,H],j∈[1,W]}, the total number of nodes is N0=H×W, where H represents the height of the deformation field, W represents the width of the deformation field, and v i,j Represents a node element;

[0039] Construct the spatial adjacency matrix A spatial and feature similarity adjacency matrix A feature ,in, POS m and pos n Indicates the position between node m and node n, radius indicates the preset distance, F m and F n Represent the fusion feature vectors of node m and node n respectively, σ f is the parameter that controls the similarity attenuation, and |·|2 represents the L2 norm;

[0040] For the spatial adjacency matrix A spatial and feature similarity adjacency matrix A feature Perform weighted summation to obtain the adjacency matrix A;

[0041] Determine the degree matrix D0 of the adjacency matrix A and the graph Laplacian matrix L;

[0042] Normalize the graph Laplace matrix L to obtain the normalized graph Laplace matrix L norm ;

[0043] A two-layer graph convolutional network is used to generate the corrected deformation field.

[0044] Furthermore, the method includes designing a total loss function as:

[0045] L total =λ1L registration +λ2L contrast +λ3L consistency +λ4L smoothness +λ5L uncertainty +λ6L kl , where L regostration represents the registration loss function, L contrast represents the contrastive learning loss function, L consistency represents the spatial consistency constraint loss function, L smoothness represents the spatial smoothness constraint loss function, L uncertainty represents the uncertainty loss function, L kl represents the divergence loss function, λ1, λ2, λ3, λ4, λ5 and λ6 represent the weights of each loss function, and the spatial smoothing loss function L smoothness =tr(D' T L norm D'), where D' represents the result after flattening the deformation field corrected by the double-layer graph convolutional network.

[0046] Furthermore, the registration loss function is: Among them, T(I r ,D) represents the deformation field D used to transform the infrared image I r Perform space transformation operations. represents the gradient operator, α is the weight coefficient of the gradient term, |·|2 represents the L2 norm, and the first term in the registration loss function ensures pixel-level similarity, while the second term ensures the consistency of edge information.

[0047] Furthermore, the uncertainty loss function is:

[0048] Among them, y i is the infrared image in the i-th visible-infrared image pair in the training data, μ i is the predicted mean value of the ith infrared image after registration, is the variance of the noise of the predicted registered ith infrared image, and N represents the total number of visible-infrared image pairs in the training data.

[0049] The beneficial effects of the present invention are:

[0050] By designing a modality-specific feature extraction network and introducing an attention mechanism and a contrastive learning strategy, this paper successfully establishes cross-modal semantic associations, making two modalities that were originally not directly comparable comparable in the feature space.

[0051] The present invention also solves the problem of large differences in resolution between images of different modalities through multi-scale feature fusion technology, thereby achieving high-precision registration;

[0052] This paper achieves stable registration of pigs in different postures and deformation states by learning semantic consistency features between different modalities and establishing a deformation field estimation model, overcoming the registration difficulties caused by non-rigid deformation of the pig body. Furthermore, by developing a progressive deformation field estimator, adopting a coarse-to-fine multi-level deformation estimation strategy, and combining a graph convolutional network to establish topological relationships between pixels, it can accurately capture large-scale posture changes while precisely describing local subtle deformations, ensuring the physical rationality and structural coherence of the deformation field.

[0053] The present invention also introduces a Bayesian deformation network, which can simultaneously predict the deformation field and its uncertainty, realize the quantitative evaluation of the reliability of the registration result, and provide an important credibility indicator for subsequent applications;

[0054] The present invention also simultaneously considers multiple mutually constrained objectives during the registration process, such as feature matching accuracy, deformation field smoothness, and cycle consistency. It designs a multi-task joint optimization framework that can coordinate multiple loss functions and automatically balance various optimization objectives through an adaptive weight learning mechanism, significantly improving the overall performance and robustness of the registration.

[0055] Therefore, the present invention solves the problem of cross-modal registration between visible light images and infrared thermal imaging, achieves high-precision pixel-level alignment, and provides a technical basis for non-contact pig temperature monitoring.

[0056] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0058] Figure 1 This is a schematic flowchart of the visible light-infrared dual-modal image registration method based on deep learning;

[0059] Figure 2 This is the architecture diagram of visible light-infrared dual-modality image registration;

[0060] Figure 3 It is a schematic flow chart of image feature extraction and enhancement;

[0061] Figure 4 It is a schematic flowchart of multi-scale feature fusion;

[0062] Figure 5 This is a schematic flowchart of deformation field estimation using a hierarchical deformation field estimation network based on deformable convolution. DETAILED DESCRIPTION

[0063] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, and are not intended to limit the scope of protection of the present invention.

[0064] The present invention proposes a visible light-infrared dual-modal image registration system and method based on deep learning. The system can be applied to the cross-modal registration technical difficulties existing in the application scenario of non-contact body temperature detection of live pigs.

[0065] The adaptive multi-scale feature fusion algorithm proposed in this invention innovatively solves the problem of significant resolution difference between visible light images (2560×1440 resolution) and infrared images (320×240 resolution). The invention realizes accurate registration between images of different resolutions by constructing a multi-scale feature pyramid, designing an adaptive weight learning mechanism, and adopting learnable modal weight parameters for cross-modal fusion. The Bayesian uncertainty modeling framework proposed in this invention introduces a joint modeling mechanism of cognitive uncertainty, accidental uncertainty and environmental uncertainty in the field of image registration for the first time. The invention uses Monte Carlo The Dropout method estimates cognitive uncertainty, the dedicated network branch predicts accidental uncertainty, and the uncertainty modeling of environmental factor perception provides a reliability quantitative assessment for the registration results; the graph convolutional network spatial constraint module in the present invention ensures the spatial continuity and smoothness of the deformation field by constructing a pixel-level graph structure, calculating a fusion adjacency matrix (combining spatial adjacency and feature similarity), and applying graph Laplace constraints, thereby avoiding non-physical local distortions; the hierarchical progressive deformation field estimation network in the present invention adopts a coarse-to-fine five-layer pyramid structure, combined with deformable convolution technology and a bidirectional consistency correction mechanism, to achieve high-precision non-rigid deformation modeling, and the target registration error is controlled within 2 pixels; moreover, the present invention adopts an end-to-end multi-task learning framework, and by jointly optimizing six objective functions such as registration loss, contrastive learning loss, consistency loss, spatial smoothness loss, uncertainty loss and KL divergence loss, it achieves collaborative optimization of feature extraction, registration estimation, and uncertainty modeling, avoiding the error accumulation problem of traditional multi-step methods.

[0066] refer to Figure 2The overall architecture of the system consists of six core subsystems that work together in a specific order to achieve accurate image registration. These subsystems are the data acquisition subsystem, the image preprocessing module, the deep learning registration network, the uncertainty modeling module, the spatial constraint module, and the post-processing and output module.

[0067] The data acquisition subsystem first collects a three-channel RGB image of the target area with a resolution of 2560×1440 pixels through a high-resolution visible light camera, which is recorded as Among them, the height of the visible light image H v =1440, width W of the visible light image v = 2560. Simultaneously, a single-channel infrared image (i.e., IRT image) of the same target area with a resolution of 320×240 pixels is collected by an infrared thermal imager and recorded as Among them, the height of the infrared image H r =240, the width of the infrared image W r = 320. The two cameras use hardware synchronization triggers to ensure the time consistency of image acquisition in the same scene, and the delay of the trigger signal is controlled within 1 millisecond.

[0068] The image preprocessing module preprocesses the collected original image to improve the image quality and perform initial alignment. After the initial alignment, the two keep the same size in width and height, such as Figure 2 As shown, the infrared image and the visible light image keep the same size in width and height. Figure 2 The image shown in is the image after initial alignment.

[0069] For visible light images, first, a Gaussian filter with a standard deviation of 1.0 can be applied to perform denoising. The filtering operation is defined as: v,clean =G σ＝1.0 *I v , (1)

[0070] Among them, * represents the convolution operation, G σ represents a Gaussian kernel with standard deviation σ, I v,clean Represents the visible light image after filtering and denoising.

[0071] Next, enhance the image contrast through histogram equalization:

[0072] I v,enhanced (i,j)=CDF(I v,clean (i,j))×(L-1), (2)

[0073] Where i and j in formula (2) represent the horizontal and vertical coordinate numbers of the pixel points in the image (i and j in other formulas may have different meanings, which will be explained in the corresponding places), CDF(·) is the cumulative distribution function, L is the number of gray levels (256 for 8-bit images), I v,enhanced Represents the visible light image after contrast enhancement.

[0074] For infrared images, first, you can apply a median filter with a kernel size of 3×3 to remove salt and pepper noise:

[0075] I r,clean (i,j)=medianI r (i+u,j+v)|(u,v)∈W, (3)

[0076] Where i and j in formula (3) represent the horizontal and vertical coordinate numbers of the pixel points in the image, W is a 3×3 neighborhood window, and I r,clean It is the infrared image after median filtering.

[0077] Then, the contrast-limited adaptive histogram equalization (CLAHE) method is used to enhance the local contrast of the infrared image after median filtering. That is, by dividing the image into several small areas, each small area is independently subjected to histogram equalization to avoid local overbrightness or overdarkness caused by global processing. This method can generate an infrared image with enhanced contrast. r,enhanced .

[0078] The denoising and equalization of the visible light image and the denoising and equalization of the infrared image can be performed in parallel or in sequence, and there is no limitation here.

[0079] Finally, the visible light image and the infrared image (i.e., the visible light image and the infrared image after denoising and equalization) are initially geometrically aligned through SIFT feature point detection and matching.

[0080] SIFT feature detection first constructs a Gaussian difference pyramid to detect key points, and then calculates a 128-dimensional feature descriptor for each key point. Let the set of key points of the detected visible light image be {kp v,i |i=1,2,...,N v}, the infrared image key point set is {kp v,j |j=1,2,...,N r}.

[0081] Matching is performed by calculating the Euclidean distance between feature descriptors: d ij =|descv,o -desc r,j |2, (4)

[0082] Among them, |·|2 represents the L2 norm, desc v,i Represents the feature descriptor of the i-th key point in the set of key points of the visible light image, desc r,j Denotes the feature descriptor of the jth key point in the infrared image key point set, d ij Represents the Euclidean distance between two descriptors.

[0083] Use the ratio test to screen for reliable matches: If d ij <0.75×d ik (where d ik is the second nearest neighbor distance, 0.75 is an exemplary ratio, and the ratio can be any suitable value, such as any suitable value between 0.7 and 0.8), then it is considered a valid match, otherwise it is considered an invalid match and the invalid match is eliminated.

[0084] Use the matching point pairs to estimate the initial homography transformation matrix H through the RANSAC algorithm init ,The matrix can be used to transform the infrared image so that it can be preliminarily geometrically aligned with the visible light image.

[0085] The deep learning registration network is the core technical module of the system, using an end-to-end deep learning architecture to accurately register visible and infrared images. Based on an encoder-decoder structure, the network combines attention mechanisms and spatial constraints to address cross-modal registration.

[0086] The uncertainty modeling module quantitatively evaluates the uncertainty of the registration results based on the Bayesian neural network, where the uncertainty includes epistemic uncertainty and aleatoric uncertainty.

[0087] The spatial constraint module establishes the spatial relationship between pixels through a graph convolutional network, imposes smoothness constraints on the deformation field, and ensures the spatial continuity of the registration results.

[0088] The post-processing and output module performs quality assessment on the registration results and generates the final temperature distribution map and confidence map.

[0089] Reference Figure 1 The present invention also provides a visible light-infrared dual-modal image registration method based on deep learning, which further includes:

[0090] Step 1: Acquire two images of the same scene in different modes, namely, a visible light image and an infrared image;

[0091] Step 2: Preprocess the visible light image and infrared image separately;

[0092] Step 3: Feature extraction is performed on the preprocessed visible light image and the preprocessed infrared image respectively, and the feature extraction results are enhanced using the dual attention mechanism to generate enhanced visible light image feature maps and enhanced infrared image feature maps. The dual attention mechanism includes channel attention and spatial attention.

[0093] Step 4: Perform multi-scale feature fusion on the enhanced visible light image features and the enhanced infrared image features to generate cross-modal fusion features;

[0094] Step 5: Input the cross-modal fusion features into the deformation field estimation network to generate an estimation result of the deformation field. The deformation field estimation network is a layered deformation field estimation network based on deformable convolution. The deformation field estimation network contains multiple layers of deformation field estimation subnetworks, and each layer of deformation field estimation subnetwork generates the deformation field estimation result corresponding to that layer.

[0095] Step 6: Apply the estimated deformation field to the infrared image to register the corresponding visible light image.

[0096] The preprocessing in step 2 is the same as that described above. For example, a Gaussian filter can be used to denoise the visible light image and histogram equalization can be used to enhance image contrast. A median filter can be used to remove salt and pepper noise from the infrared image and contrast-limited adaptive histogram equalization can be used to enhance local contrast. The preprocessing also includes preliminary geometric alignment of the enhanced visible light image and the enhanced infrared image using SIFT feature point detection and matching.

[0097] Figure 3 It is a schematic flow chart of image feature extraction and enhancement, such as Figure 3 As shown, step 3 may specifically include:

[0098] S31: Use the pre-trained ResNet-50 network to perform initial feature extraction on the pre-processed visible light image and the pre-processed infrared image, respectively, to generate a visible light image feature map and an infrared image feature map;

[0099] S32: Use a channel attention mechanism to enhance the visible light image feature map and the infrared image feature map respectively to generate a first visible light image enhanced feature map and a first infrared image enhanced feature map;

[0100] S33: Use the spatial attention mechanism to enhance the first visible light image enhancement feature map and the first infrared image enhancement feature map respectively to generate a second visible light image enhancement feature map and a second infrared image enhancement feature map, i.e., an enhanced visible light image feature map and an enhanced infrared image feature map.

[0101] The above extraction and enhancement of visible light image and infrared image features can be divided into two branches, namely, visible light feature extraction branch and infrared feature extraction branch. The image feature extraction of these two branches can be performed in parallel.

[0102] Specifically, for the visible light feature extraction branch, the preprocessed image can be normalized first:

[0103]

[0104] Among them, μ ImageNet and σ ImageNet are the mean and standard deviation vectors of the ImageNet dataset, I v,norm Represents a normalized visible light image. Since ResNet-50 is pre-trained on ImageNet, normalizing the input in this way maintains compatibility with the pre-trained weights while also leveraging pre-trained knowledge to improve feature extraction.

[0105] The normalized image is fed into the attention-enhanced ResNet-50 network designed by this invention. This network is based on the standard ResNet-50 architecture, but with a dual-attention mechanism embedded within each residual block group, forming an attention-enhanced feature extraction network.

[0106] The attention-enhanced ResNet-50 of the present invention includes four attention-enhanced residual block groups, and the detailed configurations of these four residual block groups are as follows:

[0107] Initial feature map layer: The first convolutional layer of the network uses a 7×7 convolution kernel with a stride of 2 and a padding of 3 to map the 3-channel input to a 64-channel feature map, which is then downsampled by a 3×3 maximum pooling layer (with a stride of 2).

[0108] The first attention-enhanced residual block group consists of three attention-enhanced residual units, each of which incorporates a dual-attention mechanism in parallel with the standard bottleneck structure. The standard bottleneck structure is 1×1 convolution (64 channels) → 3×3 convolution (64 channels) → 1×1 convolution (256 channels). The dual-attention mechanism operates in parallel with this bottleneck structure, adaptively modulating the output features of the bottleneck structure by calculating channel-wise and spatial-wise attention weights. Finally, the attention-enhanced feature output is obtained through weighted fusion.

[0109] The second attention-enhanced residual block group contains 4 attention-enhanced residual units, and the channel configuration of the bottleneck structure is 128, 128, and 512. Each unit also adopts the design of attention mechanism in parallel with the bottleneck structure, and calculates the feature importance weight while performing residual transformation.

[0110] The third attention-enhanced residual block group: contains 6 attention-enhanced residual units, the channel configuration of the bottleneck structure is 256, 256, 1024, and the attention mechanism performs importance weight calculation and feature modulation in a higher-dimensional feature space.

[0111] The fourth attention-enhanced residual block group: contains 3 attention-enhanced residual units, the channel configuration of the bottleneck structure is 512, 512, 2048, and the attention mechanism performs refined importance weight distribution on the highest-level semantic features.

[0112] Feature dimension compression layer: Finally, the 2048 channels are compressed to 256 channels through 1×1 convolution to obtain the feature representation for subsequent registration processing.

[0113] In each attention-enhanced residual unit, the specific implementation process of the dual attention mechanism is as follows: after the bottleneck structure completes the feature transformation, the output feature map is used as the input of the attention mechanism, the importance of the channel dimension is calculated through global average pooling and global maximum pooling, and then the channel attention weight is generated through a shared multi-layer perceptron; then the channel-weighted features are pooled in the spatial dimension, and the spatial attention weight is generated through a 7×7 convolutional layer; finally, the original bottleneck output is residually connected with the attention-weighted features to obtain an enhanced feature representation that maintains residual learning ability and has adaptive feature selection ability.

[0114] The processing flow of the infrared feature extraction branch is similar to that of the visible light branch, but when using the ResNet-50 network, the number of input channels of its initial convolutional layer is changed to 1. To process single-channel input, the pre-trained weights are averaged over the channel dimension:

[0115]

[0116] Among them, W rgb,R 、W rgb,G 、W rgb,B are the weights of the original RGB three channels respectively.

[0117] For the visible light feature extraction branch and the infrared feature extraction branch, such as Figure 2 As shown, the "residual module 1", "residual module 2", "residual module 3" and "residual module 4" in these two branches correspond to the above four attention-enhanced residual block groups respectively. Figure 2 The four attention modules on each branch represent the dual attention mechanism embedded in the corresponding residual block group. These attention mechanisms form a parallel working mode with the bottleneck structure of the residual block group, achieving adaptive selection and enhancement of features while maintaining the advantages of deep residual learning.

[0118] In the present invention, each residual block group can not only extract hierarchical feature representation, but also adaptively highlight important features and suppress redundant information according to the requirements of the cross-modal registration task, thereby providing feature input for subsequent multi-scale feature fusion.

[0119] Specifically, channel attention enhancement includes the following steps:

[0120] S321: Feature map of the input channel attention mechanism to the i-th residual block group attention mechanism Perform global average pooling and global maximum pooling operations respectively, where

[0121] The calculation formula for global average pooling is:

[0122] The calculation formula for global maximum pooling is:

[0123] In formulas (7) and (8), H i Represents the height of the feature map of the i-th residual block, W i represents the width of the feature map of the i-th residual block, C i represents the number of channels of the feature map of the i-th residual block, h and w represent the sequence numbers of the feature map in the height dimension and width dimension, represents the global average pooling result corresponding to the i-th residual block, Represents the global maximum pooling result corresponding to the i-th residual block.

[0124] In our attention-enhancing design, the dual attention mechanism for each residual block group must be adapted to the dimensions of its corresponding feature map. The four residual block groups generate feature maps of 128×128×256, 64×64×512, 32×32×1024, and 16×16×2048, respectively. The H, W, and C parameters of each attention module are adapted to the output dimensions of the residual block group to which it belongs. This multi-scale feature representation serves as the input for the subsequent multi-scale feature fusion algorithm, embodying a hierarchical feature representation from fine-grained to coarse-grained.

[0125] S322: The global average pooling result and global maximum pooling result corresponding to each residual block are input into a shared multi-layer perceptron (MLP). The MLP consists of two fully connected layers, and the number of neurons in the middle hidden layer is C / r, where C is the number of channels in the feature map and r is the compression ratio, which is set to 16 in this implementation. The calculation process of the MLP is:

[0126] Next, the pooled results are fed into a shared multi-layer perceptron (MLP). The MLP consists of two fully connected layers, and the number of neurons in the middle hidden layer is C / r, where r is the compression ratio, which can be set to 16 in this invention. The calculation process of the MLP is:

[0127] A avg =W2(ReLU(W1(F avg ))), (9-1)

[0128] A max =W2(ReLU(W1(F max ))), (9-2)

[0129] Where W1∈R C / r×C and W2∈R C×C / r It is a learnable weight matrix. It should be noted that the global average pooling results and global maximum pooling results in formulas (9-1) and (9-2) represent the pooling results corresponding to each residual block. For example, the present invention includes four residual blocks. Formulas (9-1) and (9-2) perform MLP calculations on the four global average pooling results or global maximum pooling results corresponding to these four residual blocks. The generated results also correspond to each residual block in the four residual blocks. The following formulas (10) to (16) also process the content corresponding to each residual block in the four residual blocks.

[0130] S323: Calculate channel attention weight M c , which is calculated by the following formula: M c =σ(A avg +A max ), (10)

[0131] Here, σ(·) represents the Sigmoid activation function.

[0132] S324: Determine the feature map F' after channel attention enhancement, that is: F' = F⊙M c , (11)

[0133] Where ⊙ represents element-by-element multiplication

[0134] The spatial attention mechanism is calculated based on the channel attention, which specifically includes the following steps:

[0135] S325: Feature map F'=F⊙M weighted by channel attention c Perform the pooling operation in the channel dimension, and the average pooling and maximum pooling in the channel dimension are calculated as follows:

[0136]

[0137] F' max =max c F'(:,:,c), (13)

[0138] S326: Concatenate the two pooling results in the channel dimension to obtain the concatenated result:

[0139] F' concat =[F' avg ;F' max ]∈R H×W×2 , (14)

[0140] S327: Generate spatial attention weight M through a 7×7 convolution layer s :

[0141] M s =σ(Conv 7×7 (F' concat )), (15)

[0142] Among them, σ(·) represents the Sigmoid activation function, Conv 7×7 Represents a 7×7 convolution.

[0143] S328: Generate the final feature output F out ,Right now:

[0144] F out =F⊙M c ⊙M s (16)

[0145] When the feature map enhanced is the visible light image feature, F out It represents the enhanced visible light image feature map (i.e., the enhanced visible light image feature map corresponding to the four residual blocks). Finally, the enhanced visible light image feature map generated by the four residual block groups can be recorded as F v ; When the feature map enhanced is the infrared image feature, F out It represents the enhanced infrared image feature map (i.e., the enhanced infrared image feature map corresponding to the four residual blocks). Finally, the enhanced infrared image feature map generated by the four residual block groups can be recorded as F r .

[0146] Due to the significant resolution difference between visible light images and infrared images (e.g., 2560×1440 vs. 320×240), we design an adaptive multi-scale feature fusion algorithm to handle this challenge.

[0147] Figure 4 It is a schematic flowchart of multi-scale feature fusion, combined with Figure 4 , step 4 may specifically include:

[0148] S41: Construct multi-scale feature pyramids for the enhanced visible light image feature map and the enhanced infrared image feature map (i.e. Figure 2 The multi-layer structure in the feature extraction module in , each residual block group in ResNet-50 corresponds to a scale).

[0149] Specifically, the output of each residual block group of ResNet-50 is directly used as the feature representation of different scales:

[0150]

[0151] in, and Represents the visible light image features and infrared image features of the i-th scale respectively. i represents the scale number. The larger the number, the lower the feature resolution. i takes 0, 1, 2 and 3. The spatial resolution of the feature maps of the four scales are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image respectively. vis 、Layer2_output vis 、Layer3_output vis and Layer4_output vis Respectively Figure 2 The output of residual module 1, the output of residual module 2, the output of residual module 3 and the output of residual module 4 in the visible light feature extraction branch; Layer1_output ir 、Layer2_output ir 、Layer3_output ir and Layer4_output ir Respectively Figure 2 The output of residual module 1, the output of residual module 2, the output of residual module 3, and the output of residual module 4 in the mid-infrared feature extraction branch.

[0152] Since the size of the feature maps output by residual modules 1 to 4 gradually decreases, but the number of channels gradually increases, in order to ensure the spatial accuracy of subsequent registration processing, features of different scales are fused through the feature pyramid network and restored to the same spatial resolution as the input image.

[0153] S42: performing channel unification processing on the visible light image features of each scale and the infrared image features of each scale, respectively, so that the visible light image features of different scales and the infrared image features of different scales have the same channel dimension.

[0154] First, the channel unification processing of each scale feature is performed through 1×1 convolution:

[0155]

[0156] We can first reduce the number of channels by performing a 1×1 convolution on the feature map with the lowest resolution (i.e., the feature map corresponding to residual module 4), and obtain: Then, yes and Upsample (i.e., Upsample, implemented by bilinear interpolation) to match the size of the feature map corresponding to the residual module 3, and then connect it horizontally with and Add them separately, so as to ensure that the and Have the same number of channels. After adding, we can get a new feature map and Right now:

[0157]

[0158] By analogy, new feature maps of other scales can be obtained and Right now:

[0159]

[0160] Upsample(·) represents a 2x upsampling operation, implemented through bilinear interpolation. This process is how Feature Pyramid Networks (FPN) processes image features.

[0161] S43: Align the spatial dimensions of visible light image features of different scales and infrared image features of different scales after channel unification, so that all features are aligned to a unified spatial dimension. That is, upsample the features of all scales to the same spatial dimension as the input image.

[0162]

[0163] Among them, scale_factor represents the scale factor of upsampling at each scale, Represents the visible light image features after upsampling at each scale; The processing method and The same, no further details here.

[0164] Then, features of different scales are weighted and fused through an adaptive weight learning mechanism. Specifically,

[0165] S44: For each scale, the importance weights of the visible light image features after spatial alignment and the importance weights of the infrared image features after spatial alignment at that scale are calculated by global average pooling and multi-layer perceptron, that is:

[0166]

[0167] Among them, GAP represents global average pooling, MLP represents multi-layer perceptron, Represents the importance weight of the visible light image feature at the i-th scale; Represents the importance weight of the infrared image feature at the i-th scale;

[0168] S45: Normalize the obtained importance weights of the visible light image features and the infrared image features at each scale respectively to generate the normalized weights of the visible light image features and the infrared image features at each scale, namely:

[0169] in, represents the normalized weight of the visible light image features at scale i; Normalized weight of infrared image features at the i-th scale;

[0170] S46: According to the normalized weights, the visible light image features and infrared image features of different scales are weighted fused to obtain the visible light image fusion features. and infrared image fusion features Right now:

[0171]

[0172] Among them, ⊙ represents element-by-element multiplication, and The spatial resolution of is the same as the input image, and the number of channels is 256, providing high-quality feature representation for subsequent cross-modal feature fusion.

[0173] S47: Through the learnable modal weight parameter β v and β r The cross-modal feature fusion is performed on the fusion features corresponding to the visible light image and the fusion features corresponding to the infrared image to obtain the cross-modal fusion features, namely:

[0174]

[0175] Among them, Concat represents the feature concatenation operation, and ResBlock represents the residual block processing, which is used to further enhance the expressiveness of the fusion feature. ResBlock includes two convolutional layers in sequence. Each convolutional layer contains a normalization layer and an activation function. The input of ResBlock is added to the output of the second normalization layer through a skip connection and then activated by the second activation function to obtain the output of ResBlock.

[0176] In some embodiments, the system of the present invention may further include a contrastive learning module. After processing the training data and obtaining cross-modal fusion features corresponding to the training data, the contrastive learning module constructs positive sample pairs (i.e., visible light-infrared feature pairs in the same scene) and negative sample pairs (feature pairs in different scenes that cannot be aligned), and performs contrastive loss calculation, thereby optimizing model parameters and enhancing the discriminability of cross-modal features.

[0177] In order to accurately model the non-rigid deformation of the pig body, the present invention constructs a multi-level deformation field estimation network based on deformable convolution (i.e. Figure 2 Deformation Estimation Module in

[15] . Figure 5 This is a schematic flow chart of deformation field estimation using a layered deformation field estimation network based on deformable convolution, as shown in Figure 5 As shown, step 5 may specifically include:

[0178] S51: Construct the cross-modal fusion features into a multi-layer feature pyramid structure. Each layer of features corresponds to a deformation field estimation subnetwork. The spatial resolution of each layer decreases in sequence. The layer with the highest resolution is the first layer (i.e., the finest layer, which is the original resolution), and the layer with the lowest resolution is the top layer (i.e., the coarsest layer. When the number of constructed feature pyramid layers is 5, the resolution of this layer is 1 / 16 of the original resolution). The feature P of each layer in the 5-layer feature pyramid structure is 1 / 16 of the original resolution. (l) It can be expressed as:

[0179]

[0180] This can be achieved by downsampling layer by layer, using 2×2 average pooling (AvgPool 2×2 ), the step size is 2, that is, P (l+1) =AvgPool 2×2 (P (l) ).

[0181] Deformation field estimation begins at the coarsest layer (l = 5). The coarsest layer's deformation field estimation network consists of three deformable convolutional blocks. Each block follows the following structure: deformable convolution (3×3 kernel, 256 channels) → batch normalization → ReLU activation → deformable convolution (3×3 kernel, 256 channels) → batch normalization. Finally, a 1×1 convolutional layer outputs a two-channel deformation field (corresponding to displacements in the x and y directions).

[0182] The input of the deformation field estimation network includes the current layer feature P (l) and the deformation field sampled on the previous layer (i.e., layer l+1) (if there is a previous layer), i.e., S52 and S53.

[0183] S52: For the top layer (e.g., the fifth layer in a five-layer structure, the coarsest layer), the deformation field is directly estimated using the layer features to generate the deformation field estimation result of the top layer, namely:

[0184] D (5) =DeformNet (5) (P (5) ), (31)

[0185] DeformNet (5) Represents the deformation field estimation subnetwork at layer 5.

[0186] S53: For other layers except the top layer (l<5), use the upsampling result of the layer feature and the deformation field estimation result of the previous layer (i.e., the splicing result of the two) as the deformation field estimation subnetwork of this layer (i.e., DeformNet (l) ) input, thereby obtaining the deformation field estimation result of this layer (D (l) ),Right now:

[0187] D (l) =DeformNet (l) (Concat[P (l) ,UpSample(D (l+1) )]), (32)

[0188] "UpSample" represents upsampling. The concatenated features are fed into the deformation field estimation subnetwork at this layer. The network structure is identical to the coarsest layer, but the number of input channels at each layer increases by two relative to the number of channels in the feature map. That is, the number of channels in each layer is 256, and with the addition of two channels for the deformation field, the total number becomes 258.

[0189] S54: The estimated result of the deformation field of the first layer (ie, the layer with the highest resolution) is used as the total deformation field estimation result D.

[0190] In each DeformNet (i.e. DeformNet(l) )’s deformable convolution layer adaptively adjusts the sampling position of the convolution kernel by learning the offset. Given an input feature map x (i.e., the P corresponding to each layer (l) ) and convolution kernel weight w, the calculation formula of deformable convolution is:

[0191] Among them, p represents the output position, y(p) represents the result after deformable convolution, and p k represents the standard deviation of the kth convolution kernel position, Δp k represents the additional offset learned, Δm k represents the learned modulation scalar, K represents the size of the convolution kernel, and the offset Δp k and modulation scalar Δm k It is learned from the input features through additional convolutional layers.

[0192] In order to ensure the physical rationality of the deformation field, a bidirectional consistency constraint is introduced. For the estimated forward deformation field D forward (i.e. the total deformation field D, the forward deformation field is to register the infrared image to the deformation field of the visible light image, and the reverse deformation field is to register the visible light image to the deformation field of the infrared image), and the corresponding reverse deformation field D is obtained through the inverse deformation estimation network (InverseDeformNet) backward :

[0193] D backward =InverseDeformNet(D forward ), (34)

[0194] Bidirectional consistency loss L consistency Defined as:

[0195]

[0196] in, Represents the composite operation of the deformation field. The consistency of the forward and reverse deformation fields is ensured by minimizing this loss function, where the implementation process of the inverse deformation estimation network is:

[0197] First, the mapping relationship from the deformed infrared image coordinates to the original coordinates is constructed. Then, the inverse deformation field D is obtained by resampling on a regular grid using a numerical interpolation method. backward , thus making holds, where Identity represents the zero displacement field.

[0198] In order to improve the robustness and reliability of the registration algorithm in complex environments, the present invention also develops an uncertainty modeling module based on Bayesian neural network, such as Figure 2The uncertainty modeling module in ,estimates the uncertainty (i.e., uncertainty) based on the Bayesian neural network, ,and thus generates the uncertainty map (i.e., the output of the uncertainty ,modeling module).

[0199] Bayesian uncertainty modeling is performed in the final layer of the deformation field estimation network (i.e., the first layer, which is also the layer with the highest resolution). Bayesian uncertainty includes epistemic uncertainty and aleatoric uncertainty. Bayesian uncertainty modeling includes:

[0200] The convolutional layer weight parameter θ in the first layer of the deformation field subnetwork in the deformation field estimation network is replaced by a probability distribution, whose posterior distribution is calculated according to Bayes' theorem:

[0201]

[0202] Among them, Data represents the training data, P(θ|Data) is the posterior distribution, P(Data|θ) is the likelihood function, and P(θ) is the prior distribution.

[0203] Epistemic uncertainty reflects the uncertainty of model parameters and is estimated by Monte Carlo Dropout. The specific implementation process is: keep the Dropout layer (dropout rate is 0.5) activated during the inference phase, and perform T (for example, 50) forward propagation sampling: f1(x), f2(x), ..., f T (x) where f t (x) represents the network output of the t-th sampling (i.e., the estimated result of the deformation field), and each sampling can obtain a predicted value of the deformation field.

[0204] That is, by multiple forward propagations and sampling the predicted results of the deformation field, the mean and variance of the deformation field are calculated, that is, the mean μ of the epistemic uncertainty epistemic and variance Right now:

[0205]

[0206] Accidental uncertainty reflects the noise in the data itself and is directly learned by adding an additional output branch to the network. That is, the output of the deformation field estimation subnetwork of the first layer has two independent output heads: one branch is used for deformation field prediction, which outputs the mean of the predicted deformation field; the other branch is used to predict the logarithmic variance, which shares the previous feature extraction layer with the deformation field estimation branch. The last layer of the logarithmic variance branch does not use an activation function to allow negative output values. The cross-modal fusion features are passed through the branch that predicts the logarithmic variance to generate the variance of accidental uncertainty.

[0207]

[0208] Among them, LogVarNet is a network branch specifically used to predict logarithmic variance, and the use of exponential function can ensure that the variance is positive.

[0209] Finally, the total uncertainty variance is obtained by combining the epistemic uncertainty variance and the aleatoric uncertainty variance

[0210] During the inference phase, a confidence map can be generated by determining the variance of the total uncertainty (this variance is the output of the uncertainty modeling module) to evaluate the registration quality (for example, the robustness of the entire registration process can be evaluated).

[0211] To ensure the spatial continuity and smoothness of the deformation field, the present invention introduces a spatial constraint module based on a graph convolutional network. The deep learning-based visible light-infrared dual-modal image registration method also includes using a graph convolutional network to correct the deformation field after generating the total deformation field estimation result using the deformation field estimation network, which specifically includes:

[0212] Construct a graph structure, that is, the total deformation field estimation result D∈R H×W×2 Each pixel position in the graph is defined as a node set V = {v i,j |i∈[1,H],j∈[1,W]}, the total number of nodes is N0=H×W, where H represents the height of the deformation field (i.e., the height of the image after the initial geometric alignment in the preprocessing stage), W represents the width of the deformation field (i.e., the width of the image after the initial geometric alignment in the preprocessing stage), and v i,j Represents the node element. Since each pixel position has components in the x and y directions, the number of channels in the deformation field is 2, corresponding to these two components respectively.

[0213] Establish edges between nodes that satisfy the Manhattan distance constraint. For node v i,j and v m,n , an edge is created between them if the Manhattan distance constraint is satisfied:

[0214] Wherein, E in formula (41) is the edge set, i and j represent the serial numbers of the horizontal and vertical coordinates of a node, m and n are the serial numbers of the horizontal and vertical coordinates of another node, and r is the neighborhood radius, which can be set to 2 in the present invention.

[0215] Next, construct the spatial adjacency matrix A spatial (which is defined based on the spatial distance of pixels) and the feature similarity adjacency matrix A feature (which is based on the similarity definition of feature vectors), where

[0216]

[0217] In equations (42) and (43), pos m and pos n Indicates the position between node m and node n, radius indicates the preset distance, F m and F n Represents the feature vectors of node m and node n respectively (i.e., fusion feature F final The feature vector at the corresponding spatial position, the number of channels is 256), σ f is a parameter that controls similarity decay;

[0218] Then, for the spatial adjacency matrix A spatial and feature similarity adjacency matrix A feature Perform weighted summation to obtain the adjacency matrix A, that is: A = α·A spatial +β·A feature , (44)

[0219] Among them, α and β are learnable fusion weight parameters.

[0220] Then, determine the degree matrix D0 of the adjacency matrix A and the graph Laplacian matrix L. The diagonal elements of the degree matrix are:

[0221] The Laplace matrix of the graph is: L = DA, (46)

[0222] Then, in order to avoid numerical instability, the graph Laplacian matrix L is normalized to obtain the normalized graph Laplacian matrix L norm :

[0223] L norm =D -1 / 2 H -1 / 2 , (47)

[0224] Then, a two-layer graph convolutional network is used to generate the corrected deformation field, where the forward propagation of the general graph convolutional layer is defined as:

[0225] H (l+1) =ReLU(D -1 / 2 AD -1 / 2 H (l) W0 (l) ), (48)

[0226] Among them, ReLU represents the activation function, H (l) Represents the node feature matrix of the lth layer (a matrix composed of all node features, each row represents the feature vector of a node), W0 (l) represents the weight matrix of layer l.

[0227] In the system of the present invention, a two-layer graph convolutional network is used. The first layer maps the 2D output of the deformation field to a 32D hidden representation, and the second layer maps it back to 2D:

[0228] H (1) =ReLU(L norm D'W0 (1) ), (49)

[0229] H (2) =L norm H (1) W0 (2) , (50)

[0230] Where D'∈R N×2 is the flattened deformation field (i.e., the result of flattening the total deformation field D generated by the previous estimation), W0 (1) ∈R 2×32 and W0 (2) ∈R 32×2 is a learnable weight matrix, H (2) That is the corrected deformation field.

[0231] The method further includes determining a spatial smoothness loss function to ensure the continuity and smoothness of the deformation field in space, wherein the spatial smoothness loss function L smoothness =tr(D' T L norm D'), where D' represents the result of flattening the total deformation field generated by the deformation field estimation network after correction by the two-layer graph convolutional network, and tr represents the trace operation of the matrix. This loss function encourages adjacent nodes to have similar deformation vectors, thereby ensuring spatial continuity of the deformation field.

[0232] The present invention adopts a multi-task learning training strategy to improve the overall performance by jointly optimizing multiple related tasks. The total loss function of the system L total for:

[0233] L total =λ1L registration +λ2L contrast +λ3L cnsistency +λ4L smoothness +λ5L uncertainty +λ6L kl , (51)

[0234] Among them, L registration represents the registration loss function, L contrast represents the contrastive learning loss function, L consistency represents the spatial consistency constraint loss function, L smoothness represents the spatial smoothness constraint loss function, Luncertainty represents the uncertainty loss function, L kl represents the divergence loss function, λ1, λ2, λ3, λ4, λ5 and λ6 represent the weights of each loss function,

[0235] The registration loss function measures the similarity between the registered image (i.e., the result after applying the deformation field to the infrared image) and the target image (i.e., the visible light image): The registration loss function can be expressed as:

[0236]

[0237] Among them, T(I r ,D) represents the deformation field D used to transform the infrared image I r Perform space transformation operations. represents the gradient operator, α is the weight coefficient of the gradient term, |·|2 represents the L2 norm, and the first term in the registration loss function ensures pixel-level similarity, while the second term ensures the consistency of edge information.

[0238] The uncertainty loss function is defined based on the Gaussian negative log-likelihood function as:

[0239]

[0240] Among them, y i is the infrared image in the i-th visible-infrared image pair in the training data, μ i is the predicted mean value of the ith infrared image after registration, is the variance of the noise of the predicted registered ith infrared image, and N represents the total number of visible-infrared image pairs in the training data.

[0241] In order to enhance the discriminability of cross-modal features, contrastive learning loss is introduced:

[0242]

[0243] in, and is a positive sample pair (i.e., a visible light-infrared feature pair under the same scene), is the negative sample feature (i.e., the feature that is not aligned in different scenarios), k represents the number of negative samples, and sim represents the cosine similarity function:

[0244]

[0245] Here, τ is a temperature parameter that controls the sharpness of the distribution.

[0246] KL divergence loss is used for regularization of Bayesian neural networks and calculates the KL divergence between the prior distribution and the posterior distribution of the weight parameters in the Bayesian neural network:

[0247]

[0248] Here, q(θ) is the posterior distribution of the weights, and p(θ) is the prior distribution.

[0249] After the deformation field estimation network is trained, the trained deformation field estimation network can be used to estimate the deformation field, and then the estimated deformation field can be applied to image registration, and quality assessment and temperature information extraction can be performed.

[0250] The spatial transformer is implemented based on bilinear interpolation. Given a deformation field D(i,j) = [u(i,j), v(i,j)], u(i,j) and v(i,j) describe the mapping relationship from the source image to the target image. Each position (i,j) in the target image (here refers to the infrared image after registration) corresponds to the position (i+u(i,j),j+v(i,j)) in the source image (the infrared image before registration). Since this position is usually not an integer coordinate, bilinear interpolation is used to calculate the pixel value:

[0251]

[0252] Among them, the weight w pq Calculated based on distance: w 00 =(1-α0)(1-β0),w 01 =(1-α0)β0,

[0253] w 10 =α0(1-β0),w 11 =α0β0, where Indicates rounding down.

[0254] The registration quality is evaluated using multiple metrics. The Structural Similarity Index (SSIM) is calculated as:

[0255]

[0256] Where μ1 and μ2 are the means of the two images I1 and I2, that is, the means of the registered infrared image and the original visible light image. are the variances of the two images, σ 12 is the covariance, c1 = (0.01 × 255) 2 c2 = (0.03 × 255) 2 is a stability constant.

[0257] The calculation formula of the normalized cross correlation coefficient (NCC) is:

[0258]

[0259] in, are the average grayscale values ​​of the two images respectively.

[0260] In order to meet the requirements of real-time processing, the present invention also adopts multiple performance optimization strategies.

[0261] The 32-bit floating point model is converted to an 8-bit integer model using the post-training quantization method. The quantization process includes two parts: weight quantization and activation quantization. Weight quantization determines the quantization parameters by calculating the maximum and minimum values ​​of each layer's weights:

[0262]

[0263] Among them, w max and w min Represents the maximum and minimum weights of each layer respectively, and scale represents the quantization parameter scaling factor.

[0264] The quantized weight is calculated as:

[0265] w float Indicates floating point weight, zero point Indicates zero point, w quantized represents the quantized weight, and round(·) represents the rounding operation.

[0266] Activation quantization uses the calibration dataset to statistically analyze the distribution of activation values ​​and uses the KL divergence minimization method to determine the optimal quantization threshold.

[0267] The present invention also employs a parallel processing architecture: the system employs a pipelined parallel processing architecture, breaking the registration process into multiple stages that can be executed in parallel. This is specifically implemented using Python's concurrent.futures module: the image preprocessing stage uses two threads to process the visible light and infrared images respectively; the feature extraction stage uses two threads to process the features of the two modalities respectively; the deformation field estimation stage uses four threads to process the different levels of estimation; and the Monte Carlo sampling for uncertainty modeling is executed in parallel using multiple threads.

[0268] This invention also optimizes data loading: a multi-process data loader is used to prefetch the next batch of data to hide I / O latency. Data preprocessing operations (such as image enhancement and normalization) are performed in parallel on the CPU, avoiding wasting GPU resources. Memory-mapped files are used to accelerate the loading of large datasets.

[0269] In order to verify the effectiveness of the present invention, a comprehensive experimental verification scheme was designed.

[0270] Dataset construction: A dataset containing three-way pigs was constructed, and visible light-infrared images were collected for each pig, totaling 1,000 pairs of images.

[0271] The ground truth of image registration was obtained by manually annotating key point correspondences, with 20 anatomical feature points annotated for each pair of images.

[0272] The present invention uses multiple quantitative indicators to evaluate system performance:

[0273] Structural Similarity Index (SSIM): Evaluates the overall quality of the registered image

[0274] Normalized Cross Correlation Coefficient (NCC): Evaluates the correlation between images

[0275] Target Registration Error (TRE): Evaluates the registration accuracy of key points

[0276] Mutual Information (MI): evaluates the degree of preservation of cross-modal information

[0277] Then a comparative experiment was conducted, and the results of the comparative experiment with the existing method are as follows:

[0278] Traditional SIFT+RANSAC method: SSIM = 0.67 ± 0.08, NCC = 0.72 ± 0.09, TRE = 7.8 ± 2.1 pixels, temperature MAE = 0.75 ± 0.15 °C, processing time = 145 ± 25 ms.

[0279] Mutual information-based registration method: SSIM = 0.74 ± 0.06, NCC = 0.79 ± 0.07, TRE = 6.2 ± 1.8 pixels, temperature MAE = 0.58 ± 0.12 °C, processing time = 1180 ± 200 ms.

[0280] Method of the present invention: SSIM = 0.946 ± 0.025, NCC = 0.963 ± 0.018, TRE = 1.65 ± 0.35 pixels, temperature MAE = 0.142 ± 0.028 ° C, processing time = 32 ± 5 ms.

[0281] Next, an ablation experiment was conducted. To verify the effectiveness of each module, a detailed ablation experiment was conducted, as follows:

[0282] Removing the dual attention mechanism: SSIM decreases by 0.024 and TRE increases by 0.8 pixels, indicating the importance of the attention mechanism in improving registration accuracy.

[0283] Removing multi-scale fusion: SSIM decreases by 0.031 and temperature MAE increases by 0.035°C, indicating the necessity of multi-scale processing for cross-resolution registration.

[0284] Removing uncertainty modeling: Although the registration accuracy remains basically unchanged, the calibration error increases by 0.15, which shows the value of uncertainty modeling for reliability assessment.

[0285] Removing the graph convolution constraint: SSIM drops by 0.018 and the smoothness of the deformation field decreases, proving the effectiveness of the spatial constraint.

[0286] In summary, the adaptive multi-scale feature fusion technology in the visible light-infrared image registration method of the present invention successfully solves the registration problem of the resolution difference between 2560×1440 and 320×240. Compared with traditional methods, the registration accuracy is improved by 38%, and the temperature detection accuracy is improved by 81%; the present invention realizes the reliability quantification of image registration for the first time through Bayesian uncertainty modeling, providing technical guarantee for stable application in complex environments; the calibration error of the uncertainty estimation in the present invention is controlled within 0.05, meeting the practical requirements; the spatial constraint of the graph convolution network in the present invention ensures the physical rationality of the deformation field, and the deformation field smoothness index is improved by 45%, effectively avoiding non-physical local distortion; the hierarchical progressive deformation field estimation in the present invention realizes high-precision non-rigid registration, and the target registration error (TRE) is controlled within 1.65 pixels, which is 57% higher than the existing deep learning method.

[0287] Furthermore, the end-to-end learning framework of the present invention avoids the error accumulation problem of traditional multi-step methods, improving overall system accuracy by 32%. The multi-task joint optimization strategy improves various performance indicators by collaboratively learning multiple related tasks, and registration accuracy is increased by 14%. Through systematic technological innovation, the present invention successfully solves the technical challenges faced by visible light-infrared image registration in the application of pig temperature detection, providing important technical support for the development of smart animal husbandry and promising prospects for industrial application.

[0288] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A visible light-infrared dual-modal image registration method based on deep learning, characterized in that: include: Step 1: Acquire two images of the same scene in different modes, namely, a visible light image and an infrared image; Step 2: Preprocess the visible light image and infrared image separately; Step 3: Feature extraction is performed on the preprocessed visible light image and the preprocessed infrared image respectively, and the feature extraction results are enhanced using the dual attention mechanism to generate enhanced visible light image feature maps and enhanced infrared image feature maps. The dual attention mechanism includes channel attention and spatial attention. Step 4: Perform multi-scale feature fusion on the enhanced visible light image features and the enhanced infrared image features to generate cross-modal fusion features; Step 5: Input the cross-modal fusion features into the deformation field estimation network to generate an estimation result of the deformation field. The deformation field estimation network is a layered deformation field estimation network based on deformable convolution. The deformation field estimation network contains multiple layers of deformation field estimation subnetworks, and each layer of deformation field estimation subnetwork generates the deformation field estimation result corresponding to that layer. Step 6: Apply the estimated deformation field to the infrared image to register the corresponding visible light image. Wherein, step 5 includes: S51: Construct the cross-modal fusion features into a multi-layer feature pyramid structure. The spatial resolution of each layer decreases in sequence. The layer with the highest resolution is the first layer, and the layer with the lowest resolution is the top layer. Each layer of features corresponds to a deformation field estimation subnetwork. S52: For the top layer, use the layer features to perform deformation field estimation to generate the deformation field estimation result of the top layer; S53: For each layer except the top layer, the features of the layer and the up-sampled result of the deformation field estimation result of the previous layer are used as the input of the deformation field estimation sub-network of the layer, thereby obtaining the deformation field estimation result of the layer; S54: Taking the deformation field estimation result of the first layer as the total deformation field estimation result D, The output of the deformation field estimation network also includes a branch for predicting logarithmic variance. The method also includes performing Bayesian uncertainty modeling at the highest resolution layer of the deformation field estimation network. The Bayesian uncertainty includes epistemic uncertainty and aleatoric uncertainty. The Bayesian uncertainty modeling includes: The weight parameters in the first-layer deformation field estimation subnetwork are modeled as probability distributions in a Bayesian framework, and the Dropout layer activation is kept during the inference phase. By performing multiple forward propagations and sampling the predicted results of the deformation field, the mean and variance of the deformation field, i.e., the mean and variance of the epistemic uncertainty, are calculated; The cross-modal fusion features are passed through the branch of predicting the logarithmic variance to generate the variance of accidental uncertainty; The total uncertainty variance is obtained by combining the epistemic uncertainty variance and the aleatoric uncertainty variance, and the total uncertainty variance is used to generate a confidence map to evaluate the registration quality.

2. The method for visible light-infrared dual-modal image registration based on deep learning according to claim 1, characterized in that: The preprocessing of visible light images includes denoising the visible light images using a Gaussian filter and enhancing the image contrast through histogram equalization; the preprocessing of infrared images includes removing salt and pepper noise in the infrared images using a median filter and enhancing the local contrast in the infrared images using a contrast-limited adaptive histogram equalization method. The preprocessing also includes preliminary geometric alignment of the enhanced visible light image and the enhanced infrared image through SIFT feature point detection and matching methods.

3. The method for visible light-infrared dual-modal image registration based on deep learning according to claim 1, characterized in that: Step 3 specifically includes: S31: Use the pre-trained ResNet-50 network to perform initial feature extraction on the pre-processed visible light image and the pre-processed infrared image, respectively, to generate a visible light image feature map and an infrared image feature map; S32: Use a channel attention mechanism to enhance the visible light image feature map and the infrared image feature map respectively to generate a first visible light image enhanced feature map and a first infrared image enhanced feature map; S33: Use the spatial attention mechanism to enhance the first visible light image enhancement feature map and the first infrared image enhancement feature map respectively to generate a second visible light image enhancement feature map and a second infrared image enhancement feature map, i.e., an enhanced visible light image feature map and an enhanced infrared image feature map.

4. The method for visible light-infrared dual-modal image registration based on deep learning according to claim 1, characterized in that: Step 4 specifically includes: S41: Constructing multi-scale feature pyramids for the enhanced visible light image feature map and the enhanced infrared image feature map respectively; S42: performing channel unification processing on the visible light image features of each scale and the infrared image features of each scale, respectively, so that the visible light image features of different scales and the infrared image features of different scales have the same channel dimension; S43: After the channels are unified, the visible light image features of different scales and the infrared image features of different scales are spatially aligned using bilinear interpolation, so that all features are aligned to a unified spatial size; S44: For each scale, respectively, the spatially aligned visible light image features and infrared image features of the scale are used to calculate the importance weights of the visible light image features and the importance weights of the infrared image features at the scale through global average pooling and multi-layer perceptron; S45: normalizing the importance weights of visible light image features and infrared image features at each scale respectively; S46: Using the normalized importance weights of each scale, weighted fusion is performed on the spatially aligned visible light image features and the spatially aligned infrared image features of different scales to obtain fused features corresponding to the visible light image and fused features corresponding to the infrared image; S47: Perform cross-modal feature fusion on the fusion features corresponding to the visible light image and the fusion features corresponding to the infrared image through learnable modal weight parameters to obtain cross-modal fusion features.

5. The method for visible light-infrared dual-modal image registration based on deep learning according to claim 1, characterized in that: The method further includes, after generating an estimation result of the total deformation field using the deformation field estimation network, correcting the deformation field using a graph convolutional network, specifically comprising: Construct a graph structure, that is, the total deformation field estimation result D∈R H×W×2 Each pixel position in the graph is defined as a node, and edges are established between nodes that meet the Manhattan distance constraint. The node set is V = {v i,j |i∈[1,H],j∈[1,W]}, the total number of nodes is N0=H×W, where H represents the height of the deformation field, W represents the width of the deformation field, and v i,j Represents a node element; Construct the spatial adjacency matrix A spatial and feature similarity adjacency matrix A feature ,in, POS m and pos n Indicates the position between node m and node n, radius indicates the preset distance, F m and F n Represent the fusion feature vectors of node m and node n respectively, σ f is the parameter that controls the similarity attenuation, and |·|2 represents the L2 norm; For the spatial adjacency matrix A spatial and feature similarity adjacency matrix A feature Perform weighted summation to obtain the adjacency matrix A; Determine the degree matrix D0 of the adjacency matrix A and the graph Laplacian matrix L; Normalize the graph Laplace matrix L to obtain the normalized graph Laplace matrix L norm ; A two-layer graph convolutional network is used to generate the corrected deformation field.

6. The method for visible light-infrared dual-modal image registration based on deep learning according to claim 5, characterized in that: The method includes designing a total loss function as: L total =λ1L registration +λ2L contrast +λ3L consistency +λ4L smoothness +λ5L uncertainty +λ6L kl , Among them, L registration represents the registration loss function, L contrast represents the contrastive learning loss function, L consistency represents the spatial consistency constraint loss function, L smoothness represents the spatial smoothness constraint loss function, L uncertainty represents the uncertainty loss function, L kl represents the divergence loss function, λ1, λ2, λ3, λ4, λ5 and λ6 represent the weights of each loss function, Spatial smoothness loss function L smoothness =tr(D' T L norm D'), where D' represents the result after flattening the deformation field corrected by the double-layer graph convolutional network.

7. The method for visible light-infrared dual-modal image registration based on deep learning according to claim 6, characterized in that: The registration loss function is: Among them, T(I r ,D) represents the deformation field D used to transform the infrared image I r Perform space transformation operations, I v represents a visible light image, represents the gradient operator, α is the weight coefficient of the gradient term, |·|2 represents the L2 norm, and the first term in the registration loss function ensures pixel-level similarity, while the second term ensures the consistency of edge information.

8. The method for visible light-infrared dual-modal image registration based on deep learning according to claim 7, characterized in that: The uncertainty loss function is: Among them, y i is the infrared image in the i-th visible-infrared image pair in the training data, μ i is the predicted mean value of the ith infrared image after registration, is the variance of the noise of the predicted registered ith infrared image, and N represents the total number of visible-infrared image pairs in the training data.

Images