Infrared disparity image splicing method based on small-scale superpixel set boundary fusion
Patent Information
- Application Number
- CN202611305300.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-10-20
AI Technical Summary
[0005]本发明提供一种基于小尺度超像素集边界融合的红外视差图像拼接方法，以克服现有技术拼接对齐精度不高，或过分对齐导致图像扭曲问题，以及通过红外图像像素点值作为色彩信息进行加工导致拼接后图像的成像与原始红外图像相斥
[0022]有益效果：本发明通过构建双层共享特征提取器，分别在语义层面和频域层面提取红外图像的深度特征，能够充分捕获红外图像中目标的轮廓结构信息与纹理细节信息；相比传统仅依赖单一特征层的方法，能够有效克服红外图像对比度低、纹理稀疏所导致的特征匹配困难问题，显著提升了特征表达的区分度和稳健性，为后续配准提供了高质量的特征基础；
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the fields of digital image processing and infrared imaging technology, and in particular to an infrared parallax image stitching method based on small-scale superpixel set boundary fusion. Background Technology
[0002] Infrared imaging technology, unaffected by lighting conditions, capable of operating in all weather conditions, and possessing a certain degree of smoke penetration capability, has a wide range of applications in fields such as nighttime surveillance, forest fire prevention, industrial inspection, and autonomous driving. However, limited by the physical size of the infrared detector array and the field of view constraints of the optical system, a single frame of infrared image often cannot cover a large-scale monitoring scene. Infrared image stitching technology, by registering and fusing multiple infrared images with overlapping areas, can effectively expand the imaging field of view and is a key technical means to solve the above contradictions.
[0003] In the field of homomodal image stitching, existing traditional or deep learning methods mainly focus on stitching visible light images, which perform feature description based on color information. However, with the rapid development of multimodal work, the introduction of infrared images for intelligent perception is crucial. Yet, the task of infrared image stitching has received relatively little attention from researchers. Existing widely used infrared image acquisition equipment often has low imaging resolution due to cost and technical limitations, which also illustrates the particular importance of infrared image stitching.
[0004] Existing image alignment methods often fail to achieve accurate alignment or cause image distortion due to over-alignment. Meanwhile, existing image stitching methods all rely on color information processing, while infrared imaging depends on the temperature emitted by objects. Using pixel values in infrared images as color information for processing obviously distorts the meaning of infrared images and contradicts their imaging properties. Summary of the Invention
[0005] This invention provides an infrared parallax image stitching method based on small-scale superpixel set boundary fusion, which overcomes the problems of low stitching alignment accuracy or image distortion caused by excessive alignment in existing technologies, as well as the problem that the image of the stitched image is incompatible with the original infrared image due to the processing of infrared image pixel values as color information.
[0006] To achieve the above objectives, the technical solution of the present invention is as follows: An infrared parallax image stitching method based on small-scale superpixel set boundary fusion includes: S1. Obtain the first and second infrared images to be stitched together using an infrared camera; S2. Based on the first infrared image and the second infrared image to be stitched, obtain the two-layer semantic-frequency domain fusion feature flow of the first infrared image and the two-layer semantic-frequency domain fusion feature flow of the second infrared image respectively through a two-layer shared feature extractor. S3. Based on the dual-layer semantic-frequency domain fusion feature flow of the first infrared image and the dual-layer semantic-frequency domain fusion feature flow of the second infrared image, the registered first infrared image and the second infrared image are obtained through a coarse-grained non-rigid registration-elastic registration network model. S4. Construct a superpixel set fusion decoder network to obtain a stitched infrared image based on the registered first infrared image and second infrared image.
[0007] Furthermore, the dual-layer shared feature extractor includes a multi-layer residual convolutional neural network, a frequency-spatial combined encoder, and a feature fusion layer; The multilayer residual convolutional neural network is used to obtain first CNN semantic features and second CNN semantic features based on the first infrared image and the second infrared image to be stitched together through semantic feature extraction operations. The frequency-space combined encoder is used to obtain first frequency domain features and second frequency domain features based on the first infrared image and the second infrared image to be stitched, through a frequency domain feature extraction operation. The feature fusion layer is used to obtain a two-layer semantic-frequency domain fusion feature flow of the first infrared image by channel cascading based on the first CNN semantic features and the first frequency domain features; and to obtain a two-layer semantic-frequency domain fusion feature flow of the second infrared image by channel cascading based on the second CNN semantic features and the second frequency domain features.
[0008] Furthermore, the coarse-grained non-rigid registration-elastic registration network model includes a coarse-grained non-rigid registration module and an overlapping region elastic registration module; The coarse-grained non-rigid registration module is used to obtain a linear homography transformation matrix based on the bilayer semantic-frequency domain fusion feature flow of the first infrared image and the bilayer semantic-frequency domain fusion feature flow of the second infrared image; and to obtain a first-stage coarse alignment feature flow and a second-stage coarse alignment feature flow based on the bilayer semantic-frequency domain fusion feature flow of the first infrared image and the bilayer semantic-frequency domain fusion feature flow of the second infrared image, respectively. The overlapping region elastic registration module is used to construct an alignment optimization model based on the first-stage coarse alignment feature flow and the second-stage coarse alignment feature flow, thereby obtaining the registered first infrared image and second infrared image.
[0009] Furthermore, the specific steps of the coarse-grained non-rigid registration module are as follows: S311. Based on the dual-layer semantic-frequency domain fusion feature flow of the first infrared image and the dual-layer semantic-frequency domain fusion feature flow of the second infrared image, the linear offsets of the four corner points, four boundary midpoints and the center point of the image of the first infrared image and the linear offsets of the four corner points, four boundary midpoints and the center point of the image of the second infrared image are obtained by multi-layer convolutional regression. S312. Based on the linear offsets of the four corner points, four boundary midpoints, and the center point of the first infrared image, calculate and obtain the actual positions of the four corner points, four boundary midpoints, and the center point of the first infrared image; based on the linear offsets of the four corner points, four boundary midpoints, and the center point of the second infrared image, calculate and obtain the actual positions of the four corner points, four boundary midpoints, and the center point of the second infrared image. S313. Using the actual position of the center point of the first infrared image as the vertex, and based on the actual positions of the midpoints of the four boundaries of the first infrared image, the first infrared image is divided to obtain four adjacent quadrilateral transformation regions of the first infrared image; using the actual position of the center point of the second infrared image as the vertex, and based on the actual positions of the midpoints of the four boundaries of the second infrared image, the second infrared image is divided to obtain four adjacent quadrilateral transformation regions of the second infrared image. S314. Based on the four adjacent quadrilateral transformation regions of the first infrared image and the four adjacent quadrilateral transformation regions of the second infrared image, the linear homography transformation matrix of each region is obtained through the linear transformation DLT method, and the expression is:
[0010] In the formula, Let be the linear homography transformation matrix of the region; The first infrared image The coordinates of the points; The coordinates of the corresponding point in the second infrared image; This is an intermediate result of the matrix regression process; The index of the point; In order to make The independent variable that achieves the minimum value ; The square of the L2 norm; S315. Based on the dual-layer semantic-frequency domain fusion feature flow of the first infrared image, a first-stage coarse alignment feature flow is obtained through the linear homography transformation matrix of each region; based on the dual-layer semantic-frequency domain fusion feature flow of the second infrared image, a second-stage coarse alignment feature flow is obtained through the linear homography transformation matrix of each region.
[0011] Furthermore, the specific steps of the overlapping region elastic registration module are as follows: S321. Calculate and obtain the first full 0-1 mask based on the first stage coarse alignment feature flow; calculate and obtain the second full 0-1 mask based on the second stage coarse alignment feature flow. S322. Obtain the full 0-1 mask of the image overlapping region based on the first full 0-1 mask and the second full 0-1 mask; S323. Calculate and obtain the elastic registration region based on the full 0-1 mask of the overlapping area of the image; S324. Combine the first-stage coarse alignment feature flow, the second-stage coarse alignment feature flow, and the elastic registration region into a multi-scale deformation field set; based on the first infrared image, the second infrared image, and the multi-scale deformation field set, construct an alignment optimization model, the expression of which is:
[0012] In the formula, Match the data items to the matrix; For the final deformation field; This is the second infrared image; This is the first infrared image; For alignment operations; For multi-scale deformation field collection; Regularization terms to ensure smooth deformation field; S325. Based on the alignment optimization model, obtain the second infrared image registered with the first infrared image, i.e. .
[0013] Furthermore, the superpixel set fusion decoder network is used to obtain the image overlap region based on the registered first infrared image and second infrared image; divide the image overlap region into several small-scale superpixel sets; obtain the fusion boundary line based on the small-scale superpixel sets; and obtain the final stitched image by adaptive weight fusion with the fusion boundary line as the center.
[0014] Furthermore, the specific steps for dividing the overlapping regions of the image into several small-scale superpixel sets are as follows: S41. Based on the overlapping areas of the images, obtain the gradient response map through lightweight edge-aware convolution and construct low-level edge paths; S42. Based on the low-level edge paths, construct the closed edge region and obtain the geometric center of the closed edge region; S43. Based on the pixels of the geometric center and the closed edge region, the pixel distance between the geometric center and the pixels of the closed edge region is obtained through a dynamic distance metric function. The expression is as follows:
[0015] In the formula, The pixel distance between the geometric center and the pixels of the closed edge region; The geometric center pixel; Pixels that are pixels within a closed edge region; and For temperature; and These are the normalized point coordinates; To prevent higher-order infinitesimals that are divisible by 0; , and These are learnable parameters; This is the boundary semantic response value; S44. By using the pixel distance between the geometric center and the pixels in the closed edge region, iteratively cluster the pixels in the overlapping region to obtain a small-scale superpixel set.
[0016] Furthermore, before obtaining the stitched infrared image, the coarse-grained non-rigid registration-elastic registration network model and the superpixel set fusion decoder network are trained using semi-supervised learning; the expression for the total loss function of the semi-supervised training is:
[0017] In the formula, This is the total loss function; For alignment loss; For superpixel set distance loss; To smooth the constraint loss during splicing.
[0018] Furthermore, the expression for the alignment loss is:
[0019] In the formula, This is a flag indicating whether a truthy label exists. , and These are learnable parameters; The image after deformation; For the true value image; It is an L1 norm; The expression for the superpixel set distance loss is:
[0020] In the formula, These are learnable parameters; For the first Pixel distance within a small-scale superpixel set; For the first A small-scale superpixel set; The expression for the splicing smoothing constraint loss is:
[0021] In the formula, These are learnable parameters; pixels in the overlapping and fusion region of the image Low-level edge paths; pixels in the overlapping and fusion region of the image Low-level edge paths; The gradient for stitching the images; The row index of the pixels in the overlapping and fused region of the image; The column index for the pixels in the overlapping and fused region of the image.
[0022] Beneficial effects: This invention constructs a two-layer shared feature extractor to extract depth features of infrared images at both the semantic and frequency domain levels, which can fully capture the contour structure information and texture details of targets in infrared images. Compared with traditional methods that rely on only a single feature layer, it can effectively overcome the feature matching difficulties caused by low contrast and sparse texture of infrared images, significantly improve the discriminability and robustness of feature representation, and provide a high-quality feature foundation for subsequent registration. By adopting a coarse-grained non-rigid registration-elastic registration network model, the initial alignment of the overall image structure is first achieved at the coarse-grained level, and then local fine deformation compensation is performed through the elastic registration network. This progressive registration strategy from coarse to fine not only ensures global geometric consistency, but also flexibly adapts to local nonlinear deformation caused by parallax in infrared images. It solves the problem of registration failure of traditional rigid transformation models in large parallax scenarios, and significantly improves registration accuracy and adaptability. By constructing a small-scale superpixel set boundary fusion decoder network, superpixel-level segmentation and boundary fusion processing are performed on the registered infrared image, enabling smooth transition of overlapping areas with pixel-level precision. The design of the small-scale superpixel set makes the fusion boundary fit the target edge more closely, avoiding common problems in traditional stitching methods such as stitching seams, ghosting, and brightness abrupt changes. The resulting stitched infrared image has higher visual naturalness and target integrity. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a flowchart illustrating the infrared parallax image stitching method of the present invention; Figure 2 This is a flowchart of infrared parallax image stitching in an embodiment of the present invention; Figure 3 This is a schematic diagram of the processing of the coarse-grained non-rigid registration-elastic registration network module in an embodiment of the present invention; Figure 4 This is a schematic diagram of the processing of the superpixel set fusion decoder network in an embodiment of the present invention; Figure 5 This is a comparison image of the original image and the stitched image in an embodiment of the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] This embodiment provides an infrared parallax image stitching method based on small-scale superpixel set boundary fusion, such as... Figure 1 and Figure 2 As shown, it includes: S1. Obtain the first and second infrared images to be stitched together using an infrared camera; In this embodiment, the first infrared image and the second infrared image to be stitched are obtained by infrared imaging devices from different shooting positions or different viewing angles, and the two infrared images have at least a partial overlapping area.
[0027] S2. Based on the first infrared image and the second infrared image to be stitched, obtain the two-layer semantic-frequency domain fusion feature flow of the first infrared image and the two-layer semantic-frequency domain fusion feature flow of the second infrared image respectively through a two-layer shared feature extractor. Preferably, the dual-layer shared feature extractor includes a multi-layer residual convolutional neural network, a frequency-spatial combined encoder, and a feature fusion layer; The multilayer residual convolutional neural network is used to obtain first CNN semantic features and second CNN semantic features based on the first infrared image and the second infrared image to be stitched together through semantic feature extraction operations. The frequency-space combined encoder is used to obtain first frequency domain features and second frequency domain features based on the first infrared image and the second infrared image to be stitched, through a frequency domain feature extraction operation. The feature fusion layer is used to obtain a two-layer semantic-frequency domain fusion feature flow of the first infrared image by channel cascading based on the first CNN semantic features and the first frequency domain features; and to obtain a two-layer semantic-frequency domain fusion feature flow of the second infrared image by channel cascading based on the second CNN semantic features and the second frequency domain features.
[0028] In this embodiment, the semantic feature extraction operation is performed by convolution and downsampling operations at different scales to obtain the target contour, local texture and high-level semantic information in the infrared image as CNN semantic features; The frequency domain feature extraction operation first transforms the infrared image in the frequency domain using discrete Fourier transform to obtain the spectral information of the infrared image; then, the amplitude and phase information of the spectrum are encoded respectively, and the encoding results are converted to the size corresponding to the spatial semantic features to obtain multi-scale frequency domain features.
[0029] S3. Based on the dual-layer semantic-frequency domain fusion feature flow of the first infrared image and the dual-layer semantic-frequency domain fusion feature flow of the second infrared image, the registered first infrared image and the second infrared image are obtained through a coarse-grained non-rigid registration-elastic registration network model. Preferably, the coarse-grained non-rigid registration-elastic registration network model includes a coarse-grained non-rigid registration module and an overlapping region elastic registration module; The coarse-grained non-rigid registration module is used to obtain a linear homography transformation matrix based on the bilayer semantic-frequency domain fusion feature flow of the first infrared image and the bilayer semantic-frequency domain fusion feature flow of the second infrared image; and to obtain a first-stage coarse alignment feature flow and a second-stage coarse alignment feature flow based on the bilayer semantic-frequency domain fusion feature flow of the first infrared image and the bilayer semantic-frequency domain fusion feature flow of the second infrared image, respectively. Preferably, such as Figure 3 As shown, the specific steps of the coarse-grained non-rigid registration module are as follows: S311. Based on the dual-layer semantic-frequency domain fusion feature flow of the first infrared image and the dual-layer semantic-frequency domain fusion feature flow of the second infrared image, the linear offsets of the four corner points, four boundary midpoints and the center point of the image of the first infrared image and the linear offsets of the four corner points, four boundary midpoints and the center point of the image of the second infrared image are obtained by multi-layer convolutional regression. In this embodiment, the four corner points, four boundary midpoints, and the image center point are not fixed geometric positions of the image, but semantic control points predicted based on the two-layer semantic-frequency domain fusion feature flow, reflecting the non-rigid deformation distribution in the image. The offsets of the four corner points, four boundary midpoints, and the image center point relative to the standard grid position of the image are output through multi-layer convolutional regression. The position of each control point changes dynamically with the input image content, which can adaptively capture the local structural differences caused by parallax in the infrared image, providing a more accurate partitioning basis for subsequent quadrilateral region block homography estimation. S312. Based on the linear offsets of the four corner points, four boundary midpoints, and the center point of the first infrared image, calculate and obtain the actual positions of the four corner points, four boundary midpoints, and the center point of the first infrared image; based on the linear offsets of the four corner points, four boundary midpoints, and the center point of the second infrared image, calculate and obtain the actual positions of the four corner points, four boundary midpoints, and the center point of the second infrared image. S313. Using the actual position of the center point of the first infrared image as the vertex, and based on the actual positions of the midpoints of the four boundaries of the first infrared image, the first infrared image is divided to obtain four adjacent quadrilateral transformation regions of the first infrared image; using the actual position of the center point of the second infrared image as the vertex, and based on the actual positions of the midpoints of the four boundaries of the second infrared image, the second infrared image is divided to obtain four adjacent quadrilateral transformation regions of the second infrared image. Specifically, the rules for dividing the first infrared image and the second infrared image are as follows: the center point of the image is used as the common vertex, and the four corner points of the image are used as the outer corner vertices of the four regions respectively. The midpoint of the boundary between two adjacent outer corner points is used as the middle control point of the shared edge. The four regions are: the upper left quadrilateral region enclosed by the upper left corner point, the midpoint of the upper boundary, the center point of the image, and the midpoint of the left boundary; the upper right quadrilateral region enclosed by the midpoint of the upper boundary, the upper right corner point, the midpoint of the right boundary, and the center point of the image; the lower right quadrilateral region enclosed by the center point of the image, the midpoint of the right boundary, the lower right corner point, and the midpoint of the lower boundary; and the lower left quadrilateral region enclosed by the midpoint of the left boundary, the center point of the image, the midpoint of the lower boundary, and the lower left corner point. The four regions seamlessly cover the entire image and do not overlap with each other. In this embodiment, the infrared image to be registered is divided into four adjacent quadrilateral regions according to their spatial positions by four corner points, four boundary midpoints, and the image center point; each quadrilateral region is jointly defined by the corresponding corner point, boundary midpoint, and center point; by establishing homography transformation for different regions, the non-uniform parallax that exists in the infrared image at different positions can be simulated. S314. Based on the four adjacent quadrilateral transformation regions of the first infrared image and the four adjacent quadrilateral transformation regions of the second infrared image, the linear homography transformation matrix of each region is obtained through the linear transformation DLT method, and the expression is:
[0030] In the formula, Let be the linear homography transformation matrix of the region; The first infrared image The coordinates of the points; The coordinates of the corresponding point in the second infrared image; This is an intermediate result of the matrix regression process; The index of the point; In order to make The independent variable that achieves the minimum value ; The square of the L2 norm; S315. Based on the dual-layer semantic-frequency domain fusion feature flow of the first infrared image, a first-stage coarse alignment feature flow is obtained through the linear homography transformation matrix of each region; based on the dual-layer semantic-frequency domain fusion feature flow of the second infrared image, a second-stage coarse alignment feature flow is obtained through the linear homography transformation matrix of each region.
[0031] The overlapping region elastic registration module is used to construct an alignment optimization model based on the first-stage coarse alignment feature flow and the second-stage coarse alignment feature flow, thereby obtaining the registered first infrared image and second infrared image.
[0032] Preferably, the specific steps of the overlapping region elastic registration module are as follows: S321. Calculate and obtain the first full 0-1 mask based on the first stage coarse alignment feature flow; calculate and obtain the second full 0-1 mask based on the second stage coarse alignment feature flow. S322. Obtain the full 0-1 mask of the image overlapping region based on the first full 0-1 mask and the second full 0-1 mask; Specifically, the process of calculating and obtaining the full 0-1 mask is as follows: The first-stage coarse alignment feature stream is input into a lightweight mask prediction head consisting of two 3×3 convolutional layers and one 1×1 convolutional layer. The output is normalized by the Sigmoid function and then binarized with a threshold of 0.5 to obtain the full 0-1 mask for each image: a pixel value of 1 indicates that there is valid image content at that position, and a pixel value of 0 indicates that the position is an invalid filling area. Based on the full 0-1 mask of the first infrared image and the full 0-1 mask of the second infrared image, a pixel-wise logical AND operation is performed (i.e., the area where both are 1) to obtain the full 0-1 mask of the overlapping area of the images. S323. Calculate and obtain the elastic registration region based on the full 0-1 mask of the overlapping area of the image; Specifically, the elastic registration region is defined as the spatial range marked by the full 0-1 mask of the overlapping area of the image. Fine elastic registration is performed only in this region, while the pixel coordinates of the non-overlapping areas remain unchanged by the coarse registration deformation field. S324. Combine the first-stage coarse alignment feature flow, the second-stage coarse alignment feature flow, and the elastic registration region into a multi-scale deformation field set; based on the first infrared image, the second infrared image, and the multi-scale deformation field set, construct an alignment optimization model, the expression of which is:
[0033] In the formula, Match the data items to the matrix; For the final deformation field; This is the second infrared image; This is the first infrared image; For alignment operations; For multi-scale deformation field collection; Regularization terms to ensure smooth deformation field; The multi-scale deformation field set consists of residual deformation fields predicted by the overlapping region elastic registration module at four scales: 1 / 8, 1 / 4, 1 / 2, and full resolution. The deformation fields at each scale are upsampled step by step through bilinear interpolation and then accumulated to finally obtain a fine deformation field at full resolution. The multi-scale prediction strategy helps to capture large-scale deformation at coarse scales and correct local deformation details at fine scales. S325. Based on the alignment optimization model, obtain the second infrared image registered with the first infrared image, i.e. .
[0034] S4. Construct a superpixel set fusion decoder network to obtain a stitched infrared image based on the registered first infrared image and second infrared image.
[0035] Preferably, the superpixel set fusion decoder network is used to obtain the image overlap region based on the registered first infrared image and second infrared image; divide the image overlap region into several small-scale superpixel sets; obtain the fusion boundary line based on the small-scale superpixel sets; and obtain the final stitched image by adaptive weight fusion with the fusion boundary line as the center. The size range of the small-scale superpixel set is from 10×10 pixels to 30×30 pixels.
[0036] In this embodiment, as Figure 4As shown, the registered first and second infrared images are mapped to the same coordinate system. An all-0-1 mask is used to obtain the overlapping region, which is then divided into several small-scale superpixel sets. A path search graph is constructed using the boundary nodes of the superpixel sets in the overlapping region. Dynamic programming, shortest path algorithms, or graph search algorithms are used to select the path with the minimum total cost from multiple candidate paths from the starting point to the ending point as the fusion seam line. After determining the fusion seam line, the region is expanded outwards from the fusion seam line to form a fusion region. The width of the fusion region can be adaptively adjusted based on the registration error near the seam line, the temperature gradient, and the size of the superpixel sets, ultimately resulting in a stitched image. The obtained stitched image is shown below. Figure 5 As shown.
[0037] Preferably, the specific steps for dividing the overlapping regions of the image into several small-scale superpixel sets are as follows: S41. Based on the overlapping areas of the images, obtain the gradient response map through lightweight edge-aware convolution and construct low-level edge paths; S42. Based on the low-level edge paths, construct the closed edge region and obtain the geometric center of the closed edge region; S43. Based on the pixels of the geometric center and the closed edge region, the pixel distance between the geometric center and the pixels of the closed edge region is obtained through a dynamic distance metric function. The expression is as follows:
[0038] In the formula, The pixel distance between the geometric center and the pixels of the closed edge region; The geometric center pixel; Pixels that are pixels within a closed edge region; and For temperature; and These are the normalized point coordinates; To prevent higher-order infinitesimals that are divisible by 0; , and These are learnable parameters; This is the boundary semantic response value; The expression for the boundary semantic response value is as follows:
[0039]
[0040]
[0041] In the formula, The edge response strength; The semantic response strength; These are learnable parameters; Normalize the Sigmoid function; For Gaussian kernel image with standard deviation Perform convolution; For two-dimensional Laplacian operators on images Perform second derivative; For pixels Category The softmax probability; The size of the Gaussian kernel; For image; For pixel category index; Total number of pixel categories; S44. By using the pixel distance between the geometric center and the pixels in the closed edge region, iterative clustering of the pixels in the overlapping region is performed to obtain a small-scale superpixel set; The clustering rules are as follows: If the pixel distance between the geometric center and the closed edge region pixel is less than a preset distance threshold, then the pixel is close to the center pixel and is included in the small-scale superpixel set; otherwise, the pixel is significantly different from the center pixel and is not included in the small-scale superpixel set.
[0042] Preferably, before obtaining the stitched infrared image, the coarse-grained non-rigid registration-elastic registration network model and the superpixel set fusion decoder network are trained using semi-supervised learning. In this embodiment, labeled infrared image pairs and unlabeled infrared image pairs are used for semi-supervised training. Labeled infrared image pairs account for approximately 20% of the total training data, while the remaining approximately 80% are unlabeled infrared image pairs. This ratio is only a preferred implementation and can be adjusted according to the actual number of labels. The coarse-grained non-rigid registration module is mainly optimized in an unsupervised manner, that is, the registration effect is evaluated based on the pixel differences in the overlapping region of the registered image; the overlapping region elastic registration module, based on the unsupervised registration constraints, uses labeled infrared image pairs for supervised learning to limit the degree of elastic deformation and reduce excessive deformation; the superpixel set fusion decoder network is optimized in an unsupervised manner using superpixel set distance loss and stitching smoothing loss. During training, the network parameters of the coarse-grained non-rigid registration module, the overlapping region elastic registration module, and the super-pixel set fusion decoder network are updated in reverse according to the total loss function until the total loss converges or the preset number of training rounds is reached, thus obtaining the trained infrared image stitching network.
[0043] The expression for the total loss function of the semi-supervised training is:
[0044] In the formula, This is the total loss function; For alignment loss; For superpixel set distance loss; To smooth the constraint loss during splicing.
[0045] Preferably, the expression for the alignment loss is:
[0046] In the formula, This is a flag indicating whether a truthy label exists. , and These are learnable parameters; The image after deformation; For the true value image; It is an L1 norm; The expression for the superpixel set distance loss is:
[0047] In the formula, These are learnable parameters; For the first Pixel distance within a small-scale superpixel set; For the first A small-scale superpixel set; The expression for the splicing smoothing constraint loss is:
[0048] In the formula, These are learnable parameters; pixels in the overlapping and fusion region of the image Low-level edge paths; pixels in the overlapping and fusion region of the image Low-level edge paths; The gradient for stitching the images; The row index of the pixels in the overlapping and fused region of the image; The column index for the pixels in the overlapping and fused region of the image.
[0049] In this embodiment, two campus security infrared cameras with a resolution of 640×512 are fixedly installed. There is an overlap of about 30° between the two cameras, and the parallax is about 15-20 pixels. At 22:00 at night, two infrared images of the main entrance of the monitored teaching building are simultaneously acquired as the first infrared image and the second infrared image to be stitched together. First, the first and second infrared images are input into a two-layer shared feature extractor. A multi-layer residual convolutional neural network extracts semantic features at three scales: 1 / 4, 1 / 8, and 1 / 16. A frequency-space combined encoder performs a 2D Fourier transform on the input infrared images to extract amplitude and phase spectrum features, which are then encoded into frequency domain features at the same scale as the semantic features through two convolutional layers. The semantic and frequency domain features are concatenated along the channel dimension to obtain the two-layer semantic-frequency fusion feature stream F1 (size 160×128×256) of the first infrared image and the two-layer semantic-frequency fusion feature stream F2 (size 160×128×256) of the second infrared image. Next, the feature stream is input into the coarse-grained non-rigid registration module; through a convolutional regression network, the offsets of nine control points (four corner points, four boundary midpoints, and one image center point) of the two images are predicted; based on the offset control point positions, each image is divided into four quadrilateral regions, and the homography matrix of each region is calculated using the direct linear transformation (DLT) method; these homography matrices are applied to perform a preliminary transformation on the second infrared image to obtain the coarse registration result, at which point the average registration error of the overlapping region is approximately 2.3 pixels; Then, the coarsely registered features are input into the overlapping region elastic registration module. This module first generates effective region masks for the two images through a mask prediction head, and determines the overlapping region (approximately 640×200 pixels) through a logical AND operation. Within the overlapping region, a multi-scale residual deformation field prediction network is used to predict the residual deformation field at four scales: 1 / 8, 1 / 4, 1 / 2, and full resolution, and the results are accumulated to obtain the final deformation field. An alignment optimization model is constructed, and the weights of the data terms and the smoothing terms are set. After 50 iterations of optimization, the first and second infrared images are registered. At this point, the average registration error of the overlapping region is reduced to 0.8 pixels. Finally, the two registered images are input into a superpixel set fusion decoder network. Within the overlapping region, gradient response maps are extracted using lightweight edge-aware convolution to identify 34 closed edge regions. For each closed edge region, its geometric center is calculated, and the pixels within the region are clustered using a dynamic distance metric function to obtain 34 small-scale superpixel sets with an average size of approximately 15×15 pixels. A path search graph is constructed based on the superpixel set boundaries, and the Dijkstra shortest path algorithm is used to calculate the fusion seam line, with a seam line length of 512 pixels. The fusion region is extended 8 pixels wide on both sides of the seam line as the center. Within the fusion region, adaptive fusion weights are calculated based on the distance from the pixels to the seam line, and the two registered images are weighted and fused to finally generate a panoramic infrared image with a size of 1120×512.
[0050] The present invention has the following beneficial effects: This invention constructs a two-layer shared feature extractor to extract depth features from infrared images at both the semantic and frequency domain levels, which can fully capture the contour structure information and texture details of targets in infrared images. Compared with traditional methods that rely on only a single feature layer, it can effectively overcome the feature matching difficulties caused by low contrast and sparse texture in infrared images, significantly improve the discriminability and robustness of feature representation, and provide a high-quality feature foundation for subsequent registration. By adopting a coarse-grained non-rigid registration-elastic registration network model, the initial alignment of the overall image structure is first achieved at the coarse-grained level, and then local fine deformation compensation is performed through the elastic registration network. This progressive registration strategy from coarse to fine not only ensures global geometric consistency, but also flexibly adapts to local nonlinear deformation caused by parallax in infrared images. It solves the problem of registration failure of traditional rigid transformation models in large parallax scenarios, and significantly improves registration accuracy and adaptability. By constructing a small-scale superpixel set boundary fusion decoder network, superpixel-level segmentation and boundary fusion processing are performed on the registered infrared image, enabling smooth transition of overlapping areas with pixel-level precision. The design of the small-scale superpixel set makes the fusion boundary fit the target edge more closely, avoiding common problems in traditional stitching methods such as stitching seams, ghosting, and brightness abrupt changes. The resulting stitched infrared image has higher visual naturalness and target integrity.
[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for stitching infrared parallax images based on small-scale superpixel set boundary fusion, characterized in that, include: S1. Obtain the first and second infrared images to be stitched together using an infrared camera; S2. Based on the first infrared image and the second infrared image to be stitched, obtain the two-layer semantic-frequency domain fusion feature flow of the first infrared image and the two-layer semantic-frequency domain fusion feature flow of the second infrared image respectively through a two-layer shared feature extractor. S3. Based on the dual-layer semantic-frequency domain fusion feature flow of the first infrared image and the dual-layer semantic-frequency domain fusion feature flow of the second infrared image, the registered first infrared image and the second infrared image are obtained through a coarse-grained non-rigid registration-elastic registration network model. S4. Construct a superpixel set fusion decoder network to obtain a stitched infrared image based on the registered first infrared image and second infrared image.
2. The infrared parallax image stitching method based on small-scale superpixel set boundary fusion according to claim 1, characterized in that, The dual-layer shared feature extractor includes a multi-layer residual convolutional neural network, a frequency-space combined encoder, and a feature fusion layer; The multilayer residual convolutional neural network is used to obtain first CNN semantic features and second CNN semantic features based on the first infrared image and the second infrared image to be stitched together through semantic feature extraction operations. The frequency-space combined encoder is used to obtain first frequency domain features and second frequency domain features based on the first infrared image and the second infrared image to be stitched, through a frequency domain feature extraction operation. The feature fusion layer is used to obtain a two-layer semantic-frequency domain fusion feature stream of the first infrared image by channel cascading based on the first CNN semantic features and the first frequency domain features. Based on the second CNN semantic features and the second frequency domain features, a two-layer semantic-frequency domain fusion feature stream of the second infrared image is obtained by channel cascading.
3. The infrared parallax image stitching method based on small-scale superpixel set boundary fusion according to claim 1, characterized in that, The coarse-grained non-rigid registration-elastic registration network model includes a coarse-grained non-rigid registration module and an overlapping region elastic registration module; The coarse-grained non-rigid registration module is used to obtain a linear homography transformation matrix based on the bilayer semantic-frequency domain fusion feature flow of the first infrared image and the bilayer semantic-frequency domain fusion feature flow of the second infrared image; and to obtain a first-stage coarse alignment feature flow and a second-stage coarse alignment feature flow based on the bilayer semantic-frequency domain fusion feature flow of the first infrared image and the bilayer semantic-frequency domain fusion feature flow of the second infrared image, respectively. The overlapping region elastic registration module is used to construct an alignment optimization model based on the first-stage coarse alignment feature flow and the second-stage coarse alignment feature flow, thereby obtaining the registered first infrared image and second infrared image.
4. The infrared parallax image stitching method based on small-scale superpixel set boundary fusion according to claim 3, characterized in that, The specific steps of the coarse-grained non-rigid registration module are as follows: S311. Based on the dual-layer semantic-frequency domain fusion feature flow of the first infrared image and the dual-layer semantic-frequency domain fusion feature flow of the second infrared image, the linear offsets of the four corner points, four boundary midpoints and the center point of the image of the first infrared image and the linear offsets of the four corner points, four boundary midpoints and the center point of the image of the second infrared image are obtained by multi-layer convolutional regression. S312. Based on the linear offsets of the four corner points, four boundary midpoints, and the center point of the first infrared image, calculate and obtain the actual positions of the four corner points, four boundary midpoints, and the center point of the first infrared image; based on the linear offsets of the four corner points, four boundary midpoints, and the center point of the second infrared image, calculate and obtain the actual positions of the four corner points, four boundary midpoints, and the center point of the second infrared image. S313. Using the actual position of the center point of the first infrared image as the vertex, and based on the actual positions of the midpoints of the four boundaries of the first infrared image, the first infrared image is divided to obtain four adjacent quadrilateral transformation regions of the first infrared image; using the actual position of the center point of the second infrared image as the vertex, and based on the actual positions of the midpoints of the four boundaries of the second infrared image, the second infrared image is divided to obtain four adjacent quadrilateral transformation regions of the second infrared image. S314. Based on the four adjacent quadrilateral transformation regions of the first infrared image and the four adjacent quadrilateral transformation regions of the second infrared image, the linear homography transformation matrix of each region is obtained through the linear transformation DLT method, and the expression is: In the formula, Let be the linear homography transformation matrix of the region; The first infrared image The coordinates of the points; The coordinates of the corresponding point in the second infrared image; This is an intermediate result of the matrix regression process; The index of the point; In order to make The independent variable that achieves the minimum value ; The square of the L2 norm; S315. Based on the two-layer semantic-frequency domain fusion feature flow of the first infrared image, the first-stage coarse alignment feature flow is obtained through the linear homography transformation matrix of each region. Based on the two-layer semantic-frequency domain fusion feature flow of the second infrared image, the second-stage coarse alignment feature flow is obtained through the linear homography transformation matrix of each region.
5. The infrared parallax image stitching method based on small-scale superpixel set boundary fusion according to claim 4, characterized in that, The specific steps of the overlapping region elastic registration module are as follows: S321. Calculate and obtain the first full 0-1 mask based on the first stage coarse alignment feature flow; calculate and obtain the second full 0-1 mask based on the second stage coarse alignment feature flow. S322. Obtain the full 0-1 mask of the image overlapping region based on the first full 0-1 mask and the second full 0-1 mask; S323. Calculate and obtain the elastic registration region based on the full 0-1 mask of the overlapping area of the image; S324. Combine the first-stage coarse alignment feature flow, the second-stage coarse alignment feature flow, and the elastic registration region into a multi-scale deformation field set; based on the first infrared image, the second infrared image, and the multi-scale deformation field set, construct an alignment optimization model, the expression of which is: In the formula, Match the data items to the matrix; For the final deformation field; This is the second infrared image; This is the first infrared image; For alignment operations; For multi-scale deformation field collection; Regularization terms to ensure smooth deformation field; S325. Based on the alignment optimization model, obtain the second infrared image registered with the first infrared image, i.e. .
6. The infrared parallax image stitching method based on small-scale superpixel set boundary fusion according to claim 1, characterized in that, The superpixel set fusion decoder network is used to obtain the image overlap region based on the registered first infrared image and second infrared image; divide the image overlap region into several small-scale superpixel sets; obtain the fusion boundary line based on the small-scale superpixel sets; and obtain the final stitched image by adaptive weight fusion with the fusion boundary line as the center.
7. The infrared parallax image stitching method based on small-scale superpixel set boundary fusion according to claim 6, characterized in that, The specific steps for dividing the overlapping regions of an image into several small-scale superpixel sets are as follows: S41. Based on the overlapping areas of the images, obtain the gradient response map through lightweight edge-aware convolution and construct low-level edge paths; S42. Based on the low-level edge paths, construct the closed edge region and obtain the geometric center of the closed edge region; S43. Based on the pixels of the geometric center and the closed edge region, the pixel distance between the geometric center and the pixels of the closed edge region is obtained through a dynamic distance metric function. The expression is as follows: In the formula, The pixel distance between the geometric center and the pixels of the closed edge region; The geometric center pixel; Pixels that are pixels within a closed edge region; and For temperature; and These are the normalized point coordinates; To prevent higher-order infinitesimals that are divisible by 0; , and These are learnable parameters; This is the boundary semantic response value; S44. By using the pixel distance between the geometric center and the pixels in the closed edge region, iteratively cluster the pixels in the overlapping region to obtain a small-scale superpixel set.
8. The infrared parallax image stitching method based on small-scale superpixel set boundary fusion according to any one of claims 1 to 7, characterized in that, Before obtaining the stitched infrared image, the coarse-grained non-rigid registration-elastic registration network model and the superpixel set fusion decoder network are trained using semi-supervised learning; the expression for the total loss function of the semi-supervised training is: In the formula, This is the total loss function; For alignment loss; For superpixel set distance loss; To smooth the constraint loss during splicing.
9. The infrared parallax image stitching method based on small-scale superpixel set boundary fusion according to claim 8, characterized in that, The expression for the alignment loss is: In the formula, This is a flag indicating whether a truthy label exists. , and These are learnable parameters; The image after deformation; For the true value image; It is an L1 norm; The expression for the superpixel set distance loss is: In the formula, These are learnable parameters; For the first Pixel distance within a small-scale superpixel set; For the first A small-scale superpixel set; The expression for the splicing smoothing constraint loss is: In the formula, These are learnable parameters; pixels in the overlapping and fusion region of the image Low-level edge paths; pixels in the overlapping and fusion region of the image Low-level edge paths; The gradient for stitching the images; The row index of the pixels in the overlapping and fused region of the image; The column index for the pixels in the overlapping and fused region of the image.