Cross-view image registration and fusion method and system based on pattern recognition

Abstract

This invention belongs to the field of image registration and fusion technology, specifically a cross-view image registration and fusion method and system based on pattern recognition. The method includes acquiring multi-view images, preprocessing them to obtain image pairs to be registered, extracting multi-scale feature maps to construct a feature pyramid, using a cross-attention mechanism to perform cross-modal or cross-view feature interaction to generate interactive feature maps, predicting the initial spatial correspondence between the image pairs to be registered through a contrastive learning matching network, eliminating mismatches, and obtaining the spatial correspondence. Based on the established cross-view feature matching relationship, the image pairs to be registered are geometrically transformed and aligned using the spatial correspondence to obtain registered image pairs. Based on the gradient energy and structural similarity of the registered image pairs, a registration and fusion optimization network is constructed, taking the current registered image pair as input and outputting a fused image. This invention enhances the feature discrimination ability, achieves accurate registration that conforms to scene logic, improves registration accuracy, and enhances fusion quality.

Description

Technical Field

[0001] This invention belongs to the field of image registration and fusion technology, specifically a cross-view image registration and fusion method and system based on pattern recognition. Background Technology

[0002] In scenarios such as intelligent monitoring, remote sensing mapping, 3D reconstruction, and autonomous driving, cross-viewpoint and multimodal image fusion can integrate scene information from different perspectives and sensors, improving target perception and scene understanding capabilities, and has become a key technology in the field of computer vision. Existing image registration and fusion methods mostly rely on manually designed features such as SIFT and ORB or feature extraction from simple convolutional networks. Under conditions of viewpoint distortion, illumination changes, texture repetition, and significant modal differences, the stability of feature representation is insufficient, easily leading to unstable feature matching and high mismatch rates.

[0003] Traditional registration processes often rely on single-scale features and pure geometric constraints, lacking modeling of scene semantics and topology. This makes them ill-suited for handling large viewpoint shifts and non-rigid changes, leading to inaccurate spatial correspondence estimations and significant challenges in multimodal image registration with insufficient alignment accuracy. Furthermore, conventional random sampling consensus algorithms are highly dependent on initial matching quality, resulting in incomplete mismatch removal and distortion of the geometric transformation matrix, causing misalignment, ghosting, and local distortion in the registered images. In the fusion stage, existing methods often employ simple weighted or pyramidal fusion, failing to adequately integrate image gradient details and structural consistency constraints. This often results in blurred edges, loss of detail, grayscale discontinuities, and obvious fusion artifacts. Moreover, registration and fusion are typically performed independently in separate steps, with initial registration errors directly propagating to the fusion stage, hindering end-to-end optimization.

[0004] In summary, current technologies cannot simultaneously meet the requirements of matching robustness, registration accuracy, and fusion visual quality in cross-view and multimodal scenarios, which restricts the practical application effect in complex scenarios. There is an urgent need for a cross-view image registration and fusion method and system based on pattern recognition to solve the above problems. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes a cross-view image registration and fusion method and system based on pattern recognition. This invention primarily addresses the problems of unstable feature matching, high mismatch rates, and the difficulty and complexity of multimodal image registration in traditional image fusion processes.

[0006] The technical solution adopted by this invention to solve its technical problem is: the cross-view image registration and fusion method based on pattern recognition provided by this invention, comprising:

[0007] Multi-view images of the same scene are acquired, and the multi-view images are preprocessed to obtain image pairs to be registered. Multi-scale feature maps are then extracted from these pairs to construct a feature pyramid.

[0008] Based on the feature pyramid, a cross-attention mechanism is used to perform cross-modal or cross-view feature interaction and generate interactive feature maps.

[0009] Based on the interaction feature map, the initial spatial correspondence between the image pairs to be registered is predicted by the contrastive learning matching network. The random sampling consensus algorithm is then used to remove mismatches in the initial spatial correspondence to obtain the spatial correspondence.

[0010] A scene semantic topology is constructed based on a graph structure. Cross-view feature matching relationships are established using the scene semantic topology. Geometric transformations are performed on the image pairs to be registered in combination with spatial correspondences to obtain the registered image pairs.

[0011] Based on the gradient energy and structural similarity of the registered image pairs, a registration fusion optimization network is constructed. The current registered image pair is input, and the fused image is output.

[0012] The pattern recognition-based cross-view image registration and fusion method provided by this invention includes the following steps to obtain the image pair to be registered:

[0013] Determine the target scene and acquisition boundaries, select acquisition equipment according to the requirements of the target scene, and acquire raw images according to the planned acquisition path, angle, and parameters.

[0014] The original images are numbered according to the acquisition viewpoint order or modal type, and the acquisition viewpoint, time, and equipment parameters of each image are determined to obtain multi-view images.

[0015] Preprocessed multi-view images are obtained by performing image specification unification, noise suppression, illumination and grayscale correction, and image smoothing enhancement on multi-view images.

[0016] Based on the perspective association and modality type during acquisition, images with adjacent perspectives and feature overlap rates reaching a preset threshold are selected from the preprocessed multi-view images and paired to obtain image pairs to be registered.

[0017] The cross-view image registration and fusion method based on pattern recognition provided by this invention includes the following steps in constructing a feature pyramid:

[0018] The scale-invariant feature transform algorithm is used to extract corner points, edge features and corresponding feature descriptors from the image to be registered, and then mapped back to the original image to generate a first-scale feature map.

[0019] The image to be registered is downsampled by half to obtain the sampled image scale. The mid-level features are extracted by combining the histogram of oriented gradients and the local binary mode, and mapped to the sampled image scale to obtain the second scale feature map.

[0020] The sampled image scale is downsampled by half, and a lightweight convolutional neural network is used to extract deep semantic features. The deep semantic features are then deconvolutioned to generate a third-scale feature map.

[0021] The first-scale feature map, the second-scale feature map, and the third-scale feature map are integrated in a preset order, and the initial pyramid is obtained by integrating them from top to bottom and aligning them horizontally.

[0022] For each feature map of the initial pyramid, the Laplacian operator is used for edge enhancement. The weights of each feature map are dynamically adjusted according to the viewpoint differences of the image pairs to be registered to obtain the feature pyramid.

[0023] The cross-view image registration and fusion method based on pattern recognition provided by this invention includes the following steps for generating interactive feature maps:

[0024] A strategy of scale-based fusion and bidirectional fusion is adopted to perform cross-attention feature interaction on each layer of the feature pyramid feature map.

[0025] In each feature map layer, images A and B in the image pair to be registered are divided into feature blocks of a preset pixel size. The feature block of image A is used as the query, and the feature block of image B is used as the key and value for bidirectional interaction.

[0026] The similarity between feature blocks is calculated by performing a dot product operation on the query and the key, and then the cross-attention weights are obtained by normalizing them using the Softmax function.

[0027] The cross-attention weights are weighted and summed with their corresponding values ​​to obtain the feature block-level feature map after bidirectional interaction.

[0028] The feature maps corresponding to each level of the feature pyramid are standardized in size, and then integrated using a fusion strategy of weighted summation and feature concatenation to obtain an interactive feature map.

[0029] The pattern recognition-based cross-view image registration and fusion method provided by this invention includes the following steps for obtaining the initial spatial correspondence:

[0030] The interactive feature maps are projected and L2 normalized, and the feature vectors are distributed on the hypersphere to obtain feature image A and feature image B. The cosine similarity matrix between the two is calculated using matrix operations.

[0031] For each feature point in feature image A, find the point with the highest score in the cosine similarity matrix, and combine the nearest neighbor filtering strategy to filter points in feature image B, generating an initial set of matching point pairs.

[0032] The initial set of matching point pairs is mapped back to the coordinates of the original image pairs to be registered by a downsampling factor, and then filtered according to a preset similarity threshold to obtain the initial spatial correspondence.

[0033] The pattern recognition-based cross-view image registration and fusion method provided by this invention includes the following steps for obtaining spatial correspondence:

[0034] The homography matrix model is determined to be a transformation model, and the hyperparameters of the transformation model are set according to cross-viewpoint and cross-modal scenarios.

[0035] During the iteration process, multiple pairs of matching points are randomly selected from the initial spatial correspondence. Based on the coordinates of the matching points, the transformation matrix of the transformation model is solved by direct linear transformation.

[0036] Transform all points in feature image A into the coordinate system of feature image B using a transformation matrix to obtain the predicted points.

[0037] Calculate the distance between the predicted point and the actual matching point in the feature image B, and determine that points with a distance less than a preset threshold are internal points, and points with a distance greater than the preset threshold are external points.

[0038] Count the number of interior points in the current model and determine if it is greater than the number of interior points in the historical model. If so, update and transform the model. After iterating to the maximum number of times, select the set with the most interior points as the spatial correspondence.

[0039] The pattern recognition-based cross-view image registration and fusion method provided by this invention includes the following steps for establishing cross-view feature matching relationships:

[0040] Semantic segmentation is performed on feature image A and feature image B to obtain the category label for each pixel. For each semantic region, edge detection is used to extract the contour.

[0041] By analyzing connected components, pixels with the same category of labels, adjacent pixels, and connected pixels are merged into semantic target instances. For each semantic target instance, its salient features are calculated to obtain the corresponding semantic target set.

[0042] An undirected attribute graph structure is adopted, and nodes are created for each semantic target set. Based on preset requirements, it is determined whether to establish an edge and edge feature between any two nodes.

[0043] Construct a scene semantic topology for feature image A and feature image B based on node, edge, and edge features.

[0044] The semantic topology of the scene is encoded by a graph convolutional network, and the feature information of each node and its neighboring nodes are aggregated to obtain the topology information node.

[0045] The topological information nodes are embedded into the semantic topological structure of the scene. The morphological similarity of the topological information nodes of feature image A and feature image B is calculated. Combined with the comprehensive edge similarity after filtering, a similarity matrix is ​​constructed.

[0046] The probability relaxation method is used to iteratively update the node matching confidence, and the node correspondence is generated by combining the similarity matrix.

[0047] For each pair of matching nodes in the node correspondence, fine-grained feature matching is performed in the corresponding semantic region, and topological consistency filtering is performed to obtain cross-view feature matching relationships.

[0048] The pattern recognition-based cross-view image registration and fusion method provided by this invention includes the following steps to obtain registered image pairs:

[0049] Based on cross-perspective feature matching relationship and spatial correspondence, a constrained homography transformation model is constructed. A semantic topological structure regularization term is introduced, and weighted coefficients are assigned to matching points according to semantic node categories to construct a weighted least squares optimization equation.

[0050] An overdetermined system of linear equations is constructed using all matching nodes, and the geometric transformation matrix is ​​obtained by solving it in conjunction with topological constraints.

[0051] The source image is inversely mapped using the geometric transformation matrix. The corresponding coordinates of each pixel in the target image in the source image are calculated. Then, bilinear interpolation is used to resample the pixels to obtain the spatially aligned transformed image.

[0052] Local distortion correction is performed on the transformed image to obtain the corrected transformed image. The image pair to be registered is combined with the corrected transformed image to obtain the registered image pair.

[0053] The pattern recognition-based cross-view image registration and fusion method provided by this invention includes the following steps for outputting the fused image:

[0054] The registered image pairs are normalized and denoised to extract gradient energy and structural similarity.

[0055] A basic architecture model is established using a four-stage structure of feature extraction, feature fusion, weight optimization, and image construction. A fusion loss function is constructed using gradient energy, structural similarity, and pixel-level parameters. The basic architecture model is then enhanced and trained to obtain a registration and fusion optimization network.

[0056] The processed registered image is input to the registration fusion optimization network. The network inferences to obtain a preliminary fused image, and then the optimization process is performed to obtain the final fused image.

[0057] The pattern recognition-based cross-view image registration and fusion system provided by this invention includes:

[0058] The multi-scale feature construction module is used to acquire multi-view images of the same scene, preprocess the multi-view images to obtain image pairs to be registered, and extract multi-scale feature maps from them to construct a feature pyramid.

[0059] The cross-domain interaction enhancement module is used to generate interactive feature maps by performing cross-modal or cross-view feature interactions based on feature pyramids and utilizing cross-attention mechanisms.

[0060] The topology constraint matching module is used to predict the initial spatial correspondence between image pairs to be registered based on the interaction feature map through a contrastive learning matching network. The random sampling consensus algorithm is used to remove mismatches in the initial spatial correspondence to obtain the spatial correspondence.

[0061] The geometric alignment and registration module is used to construct a scene semantic topology based on a graph structure, establish cross-view feature matching relationships using the scene semantic topology, and perform geometric transformation alignment on the image pairs to be registered in combination with spatial correspondence to obtain the registered image pairs.

[0062] The energy optimization fusion module is used to construct a registration fusion optimization network based on the gradient energy and structural similarity of the registered image pairs. It takes the current registered image pair as input and outputs the fused image.

[0063] The beneficial effects of this invention are as follows:

[0064] 1. This invention utilizes a multi-scale feature pyramid combined with a cross-attention mechanism to jointly leverage shallow details and deep semantics, significantly enhancing feature representation capabilities and improving adaptability to scenes with weak or repetitive textures. Contrastive learning matching combined with mismatch removal effectively reduces initial matching errors and improves the robustness and reliability of spatial correspondences. Introducing a scene semantic topology structure combines geometric matching with semantic structural constraints, ensuring structural consistency in cross-view matching, significantly improving geometric transformation alignment accuracy, and reducing distortion and misalignment. Weighted least squares fusion of topological regularization terms makes the transformation matrix more closely resemble the spatial relationships of the real scene, resulting in more stable registration results and more accurate edge and contour alignment. A fusion network is constructed based on gradient energy and structural similarity, achieving pixel-level adaptive weight allocation, preserving clear details of both images, eliminating fusion artifacts, and producing a fused image with high contrast, structural integrity, and a natural visual appearance. Attached Figure Description

[0065] The invention will now be further described with reference to the accompanying drawings.

[0066] Figure 1 This is a flowchart illustrating the cross-view image registration and fusion method based on pattern recognition provided in an embodiment of the present invention.

[0067] Figure 2 This is a schematic diagram of the process for obtaining registered image pairs in the pattern recognition-based cross-view image registration and fusion method provided in the embodiments of the present invention;

[0068] Figure 3 This is a schematic diagram of the modules of the cross-view image registration and fusion system based on pattern recognition provided in an embodiment of the present invention. Detailed Implementation

[0069] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0070] like Figures 1 to 3 As shown, the cross-view image registration and fusion method based on pattern recognition provided in this embodiment of the invention includes:

[0071] Multi-view images of the same scene are acquired, and the multi-view images are preprocessed to obtain image pairs to be registered. Multi-scale feature maps are then extracted from these pairs to construct a feature pyramid.

[0072] The steps to obtain the image pairs to be registered include:

[0073] Determine the target scene and acquisition boundaries, select acquisition equipment according to the requirements of the target scene, and acquire raw images according to the planned acquisition path, angle, and parameters.

[0074] Clearly define the target scene to be captured, such as urban built-up areas, mountainous terrain, industrial areas, etc., and delineate the acquisition boundaries to ensure that multi-view images can completely cover the target scene, avoiding registration failures caused by scene gaps. Simultaneously record environmental parameters of the scene, such as light intensity, weather conditions, and shooting time. Select appropriate acquisition equipment based on scene requirements, such as drones, multi-camera arrays, satellites, ground cameras, etc., and standardize equipment parameters to reduce inherent differences.

[0075] Camera parameters: uniform resolution, focal length, and exposure time.

[0076] Acquisition Angle: Plan multiple shooting angles to ensure reasonable differences in perspective and feature overlap rate of adjacent perspective images of no less than 60% to avoid feature matching failure due to low overlap.

[0077] Modal adaptation: If acquiring multimodal images, such as visible light + infrared, aerial photography + ground, it is necessary to ensure that the acquisition range of different modal images is completely consistent and the acquisition time is synchronized to reduce the impact of dynamic changes in the scene, such as vehicle movement and pedestrian movement.

[0078] The original images are numbered according to the acquisition viewpoint order or modal type, and the acquisition viewpoint, time, and equipment parameters of each image are determined to obtain multi-view images.

[0079] Preprocessed multi-view images are obtained by performing image specification unification, noise suppression, illumination and grayscale correction, and image smoothing enhancement on multi-view images.

[0080] Image standardization processing: Resolution standardization: All filtered original images are adjusted to the same resolution using bilinear interpolation for scaling. If the original image resolution is too high, downsampling can be performed first to reduce computation. If the resolution is too low, interpolation is needed to supplement pixels to ensure that image details are not lost.

[0081] Image channel unification: For multimodal images, such as visible light (3 channels) and infrared (single channel), convert single-channel images to 3 channels, or convert 3-channel images to single channels (extract grayscale channels) to ensure that the number of channels in the image pair to be registered is consistent. For color images, convert them to grayscale images uniformly.

[0082] Image cropping: The adjusted image is cropped to remove invalid areas at the image edges, ensuring that the valid areas of all images are the target scene. After cropping, all images are of completely consistent size to avoid invalid areas at the edges affecting feature matching.

[0083] Noise suppression processing: Noise type determination: The noise type of the original image is determined by analyzing the image grayscale histogram and pixel grayscale fluctuations.

[0084] Gaussian noise: Gaussian filtering is used to smooth pixel values ​​by weighted averaging, which suppresses noise while preserving image edge details and avoids feature blurring caused by over-smoothing.

[0085] Salt and pepper noise: Median filtering is used to replace abnormal pixels by taking the median of the pixel's neighborhood, effectively removing salt and pepper noise without damaging the image edge structure.

[0086] Mixed noise: Gaussian-median joint filtering is used. First, Gaussian noise is suppressed by Gaussian filtering, and then salt-and-pepper noise is removed by median filtering, which balances the denoising effect and the preservation of image details.

[0087] Post-denoising verification: Compare the grayscale values ​​of the denoised image and calculate the peak signal-to-noise ratio (PSNR) before and after denoising. Ensure that the PSNR is improved by ≥3dB, which indicates that the noise has been effectively suppressed and that the image details are not significantly blurred.

[0088] Illumination and Gray Scale Correction: Illumination Difference Analysis: Calculate the mean and variance of gray scale for each image, compare the gray scale distribution of images from different viewpoints, and identify images with uneven illumination.

[0089] Global illumination correction: Histogram equalization is used to adjust the grayscale histogram distribution of the image, making the grayscale range of the image more uniform, improving image contrast, and eliminating the problem of excessively dark or bright global illumination. For images with small illumination differences, gamma correction is used to fine-tune the grayscale distribution and avoid over-correction that could lead to loss of detail.

[0090] Local illumination correction: To address uneven local illumination in the image, adaptive histogram equalization (CLAHE) is used to correct the illumination in image blocks, ensuring grayscale consistency in local areas while avoiding noise amplification caused by global equalization.

[0091] Consistency check after correction: After correction, ensure that the mean and variance of gray levels of all images are within the same range, the gray level distribution tends to be consistent, and there are no obvious differences in brightness.

[0092] Image Smoothing and Enhancement: Smoothing Processing: Bilateral filtering is used to smooth noise while preserving edge details of the image, ensuring clear image texture.

[0093] Feature enhancement: The Laplacian or Sobel operator is used to enhance the image's edges, contours, and other core features, improving feature discriminative power and providing clearer feature basis for subsequent multi-scale feature extraction and matching. After enhancement, the image is normalized to ensure a uniform range of pixel values, facilitating subsequent feature calculations.

[0094] Based on the perspective association and modality type during acquisition, images with adjacent perspectives and feature overlap rates reaching a preset threshold are selected from the preprocessed multi-view images and paired to obtain image pairs to be registered.

[0095] Based on the viewing angle association and modal type at the time of acquisition, images with adjacent viewing angles and a feature overlap rate ≥60% are preferentially selected for pairing. For example, viewing angle 1 and viewing angle 2, or viewing angle 2 and viewing angle 3. If it is cross-modal registration, images of different modalities acquired at the same time and within the same acquisition range are selected for pairing, such as visible light images and infrared images.

[0096] The steps to construct a feature pyramid include:

[0097] The scale-invariant feature transform algorithm is used to extract corner points, edge features and corresponding feature descriptors from the image to be registered, and then mapped back to the original image to generate a first-scale feature map.

[0098] An improved SIFT (Scale Invariant Feature Transform) algorithm is used to adapt to the viewpoint distortion characteristics of cross-viewpoint images. The specific operation is as follows: Scale space construction: Gaussian difference pyramids (DoG) are constructed for images A and B in the image pair to be registered. The Gaussian kernel scale σ is set to 0.5, 1.0 and 2.0, and the ratio of the standard deviation of the Gaussian kernels of adjacent scales is √2 to ensure that the detailed features of the original scale are covered.

[0099] Corner and edge detection: Local extreme points (corners) and edge points are detected in each layer of the difference of Gaussian pyramid. Invalid feature points with weak responses are removed by threshold filtering (extreme point response value ≥ 0.03).

[0100] Feature descriptor generation: For each selected feature point, a 128-dimensional SIFT descriptor is extracted. The orientation of the descriptor adopts the gradient orientation histogram (0°-360°, 10° interval) to ensure that the feature descriptor has rotation invariance and scale invariance, and adapts to feature changes across viewpoints.

[0101] The image to be registered is downsampled by half to obtain the sampled image scale. The mid-level features are extracted by combining the histogram of oriented gradients and the local binary mode, and mapped to the sampled image scale to obtain the second scale feature map.

[0102] A joint extraction method using HOG (Histogram of Oriented Gradients) and LBP (Local Binary Pattern) is employed to balance structural and textural features.

[0103] HOG feature extraction: The downsampled image is divided into 8×8 pixel cells, and each cell is divided into 4 directional blocks. The gradient histogram of each directional block is calculated to generate HOG feature vectors and capture the contour structure of the image.

[0104] LBP feature extraction: The LBP operator with a 3×3 neighborhood is used for each cell to extract local texture features. The center pixel is compared with the neighboring pixels to generate an 8-bit binary code, which is then converted into a decimal LBP value to capture local texture changes in the image.

[0105] Feature fusion: The HOG feature vector and LBP feature value are concatenated and normalized to obtain the mid-level features, which contain both structural and texture information.

[0106] The sampled image scale is downsampled by half, and a lightweight convolutional neural network is used to extract deep semantic features. The deep semantic features are then deconvolutioned to generate a third-scale feature map.

[0107] Lightweight convolutional neural networks (CNNs), such as MobileNetV3, are used to avoid slow inference caused by excessive parameters, and to meet the real-time requirements of cross-view registration. Specific operations include:

[0108] Network setup: A lightweight CNN with 4 convolutional blocks is built. Each convolutional block consists of a convolutional layer (3×3 convolutional kernel), a batch normalization layer, and a ReLU activation function. Finally, a global average pooling layer is used to output the deep feature vector.

[0109] Feature Training and Extraction: The CNN is pre-trained using a cross-view image dataset to ensure that the network can learn cross-view invariant semantic features. Images A and B, downsampled by 1 / 4, are input into the pre-trained CNN to extract the global feature vector of the last layer, capturing the global semantic information of the image, such as the semantic categories of objects like buildings, roads, and vegetation.

[0110] The first-scale feature map, the second-scale feature map, and the third-scale feature map are integrated in a preset order, and the initial pyramid is obtained by integrating them from top to bottom and aligning them horizontally.

[0111] Layer 1: Corresponds to the scale 1 feature map, which provides fine local detail features for small-scale feature matching and local alignment.

[0112] Layer 2: Corresponds to scale 2 feature map, which provides mid-level structural features for mid-scale feature interaction and global coarse matching.

[0113] Layer 3: Corresponds to the scale 3 feature map, which provides deep semantic features for global semantic constraints and matching optimization.

[0114] A "top-down fusion + horizontal alignment" approach is used to integrate feature maps from three scales into a complete feature pyramid, ensuring that features at different levels can work synergistically. Specific steps are as follows:

[0115] Top-level feature upsampling: The third-layer feature map is upsampled by a factor of 2 using bilinear interpolation to obtain a feature map with the same size as the second-layer feature map, which is denoted as the 3-1 layer feature map.

[0116] Mid-level feature fusion: The upsampled 3-1 layer feature map is added element-wise with the 2nd layer feature map, with each weight being 0.5. The deep semantic features and mid-level structural features are fused to obtain the fused mid-level feature map, denoted as 2-1 layer, which improves the semantic expressive power of the mid-level features.

[0117] Upsampling after fusion: The fused 2-1 layer feature map is upsampled again by a factor of 2 to obtain a feature map with the same size as the 1st layer feature map, which is denoted as the 2-2 layer feature map.

[0118] Low-level feature fusion: The upsampled 2-2 layer feature map is added element-wise with the 1 layer feature map, with weights of 0.3 and 0.7 respectively. The mid-level structure, deep semantic features and low-level detail features are fused to obtain the fused low-level feature map, which is denoted as 1-1 layer, ensuring that the low-level features contain both detail and semantic information.

[0119] Pyramid Assembly: The fused feature maps of layers 1-1, 2-1, and 3 are assembled in the order of "bottom layer at the bottom, top layer at the top" to form a complete feature pyramid. Among them, the bottom feature map has the highest resolution and the richest details, the top feature map has the lowest resolution and the most prominent semantics, and the middle feature map serves as a transition to connect details and semantics.

[0120] For each feature map of the initial pyramid, the Laplacian operator is used for edge enhancement. The weights of each feature map are dynamically adjusted according to the viewpoint differences of the image pairs to be registered to obtain the feature pyramid.

[0121] For each layer of the pyramid feature map, the Laplacian operator is used to enhance the edges, highlight the edge contours of the features, and improve the distinguishability of the features.

[0122] Based on the perspective differences of the image pairs to be registered, such as pitch and azimuth differences, the weights of features at each layer are dynamically adjusted: if the perspective difference is large, the weight of the top-level semantic features is increased to 0.6, relying on semantic constraints to achieve matching. If the perspective difference is small, the weight of the bottom-level detail features is increased to 0.8, relying on detail matching to improve accuracy.

[0123] Based on the feature pyramid, a cross-attention mechanism is used to perform cross-modal or cross-view feature interaction and generate interactive feature maps.

[0124] The steps for generating interaction feature maps include:

[0125] A strategy of scale-based fusion and bidirectional fusion is adopted to perform cross-attention feature interaction on each layer of the feature pyramid feature map.

[0126] In each feature map layer, images A and B in the image pair to be registered are divided into feature blocks of a preset pixel size. The feature block of image A is used as the query, and the feature block of image B is used as the key and value for bidirectional interaction.

[0127] The similarity between feature blocks is calculated by performing a dot product operation on the query and the key. This is then normalized using the Softmax function to obtain the cross-attention weights, expressed as follows:

[0128]

[0129]

[0130] In the formula, It is the attention weight of image A on image B. It is the attention weight of image B on image A. This is the query matrix for image A. This is the query matrix for image B. It is the key matrix of image A. It is the key matrix of image B. It is the number of channels for the feature. Is it a weight matrix? OK List, It is the matrix transpose.

[0131] The cross-attention weights are weighted and summed with their corresponding values ​​to obtain the feature map at the feature block level after bidirectional interaction. The formula for weighted summation is expressed as:

[0132]

[0133]

[0134] In the formula, It is an aggregate feature of image A. It is an aggregation feature of image B. It is the value matrix of image A. It is the value matrix of image B.

[0135] The formula for obtaining the hierarchical feature map is expressed as follows:

[0136]

[0137] In the formula, It is a hierarchical feature map. It is a reshaping operation that adjusts the shape of a matrix to the specified dimensions.

[0138] The feature maps corresponding to each level of the feature pyramid are standardized in size, and then integrated using a fusion strategy of weighted summation and feature concatenation to obtain an interactive feature map.

[0139] Uniform feature map size: The top-level interactive feature map is enlarged to match the size of the middle-level interactive feature map.

[0140] The bottom-level interactive feature map is downsampled by 2 times and then reduced to 128 dimensions by 1×1 convolution, which is consistent with the number of channels in the middle and top-level interactive feature maps.

[0141] Weighted summation: Based on the importance of features at each level, assign fusion weights, such as 0.4 for bottom-level detail features, 0.3 for mid-level structural features, and 0.3 for top-level semantic features. The formula is as follows: Fusion feature map = 0.4 × bottom-level interaction feature map (after downsampling) + 0.3 × mid-level interaction feature map + 0.3 × top-level interaction feature map (after upsampling).

[0142] Feature concatenation: Key regions, such as edges and target semantic regions, of the fused feature map and the interactive feature maps at each level are concatenated to supplement key feature information and enhance the richness of the interactive feature map.

[0143] Based on the interaction feature map, the initial spatial correspondence between the image pairs to be registered is predicted by the contrastive learning matching network. The random sampling consensus algorithm is then used to remove mismatches in the initial spatial correspondence to obtain the spatial correspondence.

[0144] The steps to obtain the initial spatial correspondence include:

[0145] The interactive feature maps are projected and L2 normalized, and the feature vectors are distributed on a hypersphere to obtain feature image A and feature image B. The cosine similarity matrix between the two is then calculated using matrix operations, expressed by the following formula:

[0146]

[0147] In the formula, It is a cosine similarity matrix.

[0148] For each feature point in feature image A, find the point with the highest score in the cosine similarity matrix, and combine the nearest neighbor filtering strategy to filter points in feature image B, generating an initial set of matching point pairs.

[0149] The mutual nearest neighbor strategy. That is, the point pair (i,j) is determined to be the initial spatial correspondence only if i in feature image A is the nearest neighbor of j in feature image B, and j is also the nearest neighbor of i.

[0150] The initial set of matching point pairs is mapped back to the coordinates of the original image pairs to be registered by a downsampling factor, and then filtered according to a preset similarity threshold to obtain the initial spatial correspondence.

[0151] The steps to obtain spatial correspondence include:

[0152] The homography matrix model is determined to be a transformation model, and the hyperparameters of the transformation model are set according to cross-viewpoint and cross-modal scenarios.

[0153] Hyperparameters include: Reprojection error threshold: used to determine whether a pair of points conforms to the model. For example, it can be set to 3 pixels or 5 pixels.

[0154] Maximum number of iterations: The maximum number of loops the algorithm can run to prevent infinite loops.

[0155] Minimum sample set size: The minimum number of point pairs required to solve the geometric transformation model.

[0156] During the iteration process, multiple pairs of matching points are randomly selected from the initial spatial correspondence. Based on the coordinates of the matching points, the transformation matrix of the transformation model is solved by direct linear transformation.

[0157] Transform all points in feature image A into the coordinate system of feature image B using a transformation matrix to obtain the predicted points.

[0158] Calculate the distance between the predicted point and the actual matching point in the feature image B, and determine that points with a distance less than a preset threshold are internal points, and points with a distance greater than the preset threshold are external points.

[0159] Count the number of interior points in the current model and determine if it is greater than the number of interior points in the historical model. If so, update and transform the model. After iterating to the maximum number of times, select the set with the most interior points as the spatial correspondence.

[0160] A scene semantic topology is constructed based on a graph structure. Cross-view feature matching relationships are established using the scene semantic topology. Geometric transformations are performed on the image pairs to be registered in combination with spatial correspondences to obtain the registered image pairs.

[0161] The steps to establish cross-perspective feature matching relationships include:

[0162] Semantic segmentation is performed on feature image A and feature image B to obtain the category label for each pixel. For each semantic region, edge detection is used to extract the contour.

[0163] By analyzing connected components, pixels with the same category of labels, adjacent pixels, and connected pixels are merged into semantic target instances. For each semantic target instance, its salient features are calculated to obtain the corresponding semantic target set.

[0164] Significant features include: center coordinates, bounding rectangle, area, aspect ratio, gradient direction distribution, and texture entropy.

[0165] An undirected attribute graph structure is adopted, and nodes are created for each semantic target set. Based on preset requirements, it is determined whether to establish an edge and edge feature between any two nodes.

[0166] Preset requirements include: whether the distance between any two nodes is less than a preset node threshold, and whether the spatial relationship is adjacent / parallel / enclosed, etc. Edge features include: Euclidean distance, relative azimuth angle, category compatibility (e.g., building-road compatibility, building-sky incompatibility), and topological relationships (e.g., adjacent, contained, parallel, perpendicular, intersecting).

[0167] Construct a scene semantic topology for feature image A and feature image B based on node, edge, and edge features.

[0168] The semantic topology of the scene is encoded using a graph convolutional network. The topology information nodes are obtained by aggregating the feature information of each node's neighboring nodes, as expressed by the formula:

[0169]

[0170] In the formula, It is a node Features It is its neighboring node. It is the first layer, It is the first Layer nodes eigenvectors, It is the normalization coefficient. It is the first The weight matrix of the layer, It is an activation function.

[0171] The topological information nodes are embedded into the semantic topological structure of the scene. The morphological similarity of the topological information nodes of feature image A and feature image B is calculated. Combined with the comprehensive edge similarity after filtering, a similarity matrix is ​​constructed.

[0172] The formula for calculating morphological similarity is expressed as:

[0173]

[0174] In the formula, , It is the normalized scale feature of topological information nodes. It is a shape With shape similarity, It is based on a specific set of features. It is a natural exponential function. It is the Gaussian kernel standard deviation.

[0175] The steps for calculating the overall similarity after filtering include: for any edge in the semantic topology of scene A, only consider the edges in the semantic topology of scene B that have an initial matching probability with the connected node that reaches a preset standard threshold.

[0176] The formula for calculating the combined edge similarity after filtering is expressed as:

[0177]

[0178] In the formula, It is the Euclidean distance difference between the two sides. It is the difference in azimuth between the two sides. These are weighting coefficients. It is the topological similarity. It is a combination of edge similarity. It is the maximum distance of all edges. It is angle normalization.

[0179] The probability relaxation method is used to iteratively update the node matching confidence, and the node correspondence is generated by combining the similarity matrix.

[0180] Using morphological similarity as the initial matching confidence score, the confidence score is iteratively updated, as expressed by the formula:

[0181]

[0182] In the formula, It is the first In each iteration, the confidence level of a topology information node matching another topology information node. It is the first In each iteration, the matching confidence between the neighboring nodes of the topology information node is determined.

[0183] After the iteration reaches the maximum number of iterations, the iteration stops. From the confidence matrix after the iteration converges, the node with the highest confidence is selected as the node correspondence.

[0184] For each pair of matching nodes in the node correspondence, fine-grained feature matching is performed in the corresponding semantic region, and topological consistency filtering is performed to obtain cross-view feature matching relationships.

[0185] Determining the node to which a matching point belongs: For each pair of matching points, determine the semantic target node to which it belongs.

[0186] The matching point pair satisfies the "node matching consistency". Otherwise, the matching point pair is directly removed, as it is an incorrect match due to semantic category mismatch.

[0187] For matching point pairs that satisfy the node matching consistency, further verify whether their relative spatial relationship conforms to the edge characteristics of the topological graph:

[0188] Calculate the relative azimuth of the matching point pair and the edge azimuth of its corresponding node pair.

[0189] If the azimuth deviation exceeds 30°, the matching point pair is considered to not conform to the topological spatial relationship and is therefore rejected.

[0190] The steps to obtain the registered image pairs include:

[0191] Based on cross-perspective feature matching relationship and spatial correspondence, a constrained homography transformation model is constructed. A semantic topological structure regularization term is introduced, and weighted coefficients are assigned to matching points according to semantic node categories to construct a weighted least squares optimization equation.

[0192] The formula for the constrained homography transformation model is expressed as follows:

[0193]

[0194] In the formula, These are the coordinates of a feature point in feature image A. These are the coordinates of the corresponding feature points in feature image B. It is a homography matrix.

[0195] The formula for the weighted least squares optimization equation is as follows:

[0196]

[0197] In the formula, It is the objective function. It's a real difference in angle. It's a difference in the angle predicted by the model. It is the actual distance difference. It is the distance difference predicted by the model.

[0198] An overdetermined system of linear equations is constructed using all matching nodes, and the geometric transformation matrix is ​​obtained by solving it in conjunction with topological constraints.

[0199] The formula for the overdetermined linear equation system is as follows:

[0200]

[0201] In the formula, , , , , , , , These are elements of the homography matrix.

[0202] The steps to solve for the geometric transformation matrix include:

[0203] Assign weights to each matching point and construct a weighted least squares algorithm, expressed by the formula:

[0204]

[0205] In the formula, It is the optimal parameter vector to be solved. It is to find the variable that minimizes the following expression. It is the number of matching points. It is the first The weights of each matching pair. It is the target vector. It is the parameter vector to be optimized. It is the first Design matrix of matching points, The regularization coefficients are used to find the closed-form solution of weighted least squares, expressed by the following formula:

[0206]

[0207] In the formula, It is the global design matrix. It is a diagonal weight matrix. It is the transpose of the design matrix. It is the identity matrix. It is the global observation vector.

[0208] The elements obtained from the solution are filled into the overdetermined linear equation system to obtain the geometric transformation matrix.

[0209] The source image is inversely mapped using the geometric transformation matrix. The corresponding coordinates of each pixel in the target image in the source image are calculated. Then, bilinear interpolation is used to resample the pixels to obtain the spatially aligned transformed image.

[0210] Local distortion correction is performed on the transformed image to obtain the corrected transformed image. The image pair to be registered is combined with the corrected transformed image to obtain the registered image pair.

[0211] Based on the gradient energy and structural similarity of the registered image pairs, a registration fusion optimization network is constructed. The current registered image pair is input, and the fused image is output.

[0212] The steps for outputting the fused image include:

[0213] The registered image pairs are normalized and denoised to extract gradient energy and structural similarity.

[0214] The Sobel operator is used to calculate the horizontal and vertical gradients of the two images to obtain their respective gradient energies. At the same time, the structural similarity between the two registered images is calculated region by region using a sliding window to generate structural similarity.

[0215] A basic architecture model is established using a four-stage structure of feature extraction, feature fusion, weight optimization, and image construction. A fusion loss function is constructed using gradient energy, structural similarity, and pixel-level parameters. The basic architecture model is then enhanced and trained to obtain a registration and fusion optimization network.

[0216] The preprocessed registered image and gradient energy map are concatenated and input into the network. Feature extraction is performed through multiple convolutional layers. Then, pixel-level adaptive fusion weights are generated using 1×1 convolutions and sigmoid activation. The features of the two images are weighted and fused according to the weights. Finally, a preliminary fused image is output through a reconstruction convolutional layer. The network training uses a joint loss function, which simultaneously constrains the pixel fidelity, gradient energy preservation, and structural similarity of the fused image. The Adam optimizer is used to complete the training and save the optimal model.

[0217] The processed registered image is input to the registration fusion optimization network. The network inferences to obtain a preliminary fused image, and then the optimization process is performed to obtain the final fused image.

[0218] The optimization process includes: Gray-scale equalization: Adaptive histogram equalization is performed on the initial fused image to enhance image contrast and improve detail clarity. The CLAHE parameter is set to: block size 8×8, contrast limit 0.02, to avoid excessive enhancement that could lead to noise amplification.

[0219] Edge enhancement: The Laplacian operator is used to enhance the edges of the fused image, thereby improving edge details and image sharpness.

[0220] Noise suppression: Median filtering (window size 3×3) is used to slightly reduce noise in the edge-enhanced image, removing noise amplified during the enhancement process and preserving edge details.

[0221] Pixel cropping: The pixel values ​​of the post-processed image are cropped to [0, 255] to avoid pixel value overflow.

[0222] Based on the same general inventive concept, this invention also protects a pattern recognition-based cross-view image registration and fusion system, comprising:

[0223] The multi-scale feature construction module is used to acquire multi-view images of the same scene, preprocess the multi-view images to obtain image pairs to be registered, and extract multi-scale feature maps from them to construct a feature pyramid.

[0224] The cross-domain interaction enhancement module is used to generate interactive feature maps by performing cross-modal or cross-view feature interactions based on feature pyramids and utilizing cross-attention mechanisms.

[0225] The topology constraint matching module is used to predict the initial spatial correspondence between image pairs to be registered based on the interaction feature map through a contrastive learning matching network. The random sampling consensus algorithm is used to remove mismatches in the initial spatial correspondence to obtain the spatial correspondence.

[0226] The geometric alignment and registration module is used to construct a scene semantic topology based on a graph structure, establish cross-view feature matching relationships using the scene semantic topology, and perform geometric transformation alignment on the image pairs to be registered in combination with spatial correspondence to obtain the registered image pairs.

[0227] The energy optimization fusion module is used to construct a registration fusion optimization network based on the gradient energy and structural similarity of the registered image pairs. It takes the current registered image pair as input and outputs the fused image.

[0228] In summary, the pattern recognition-based cross-view image registration and fusion method and system provided in this embodiment achieves comprehensive alignment from the pixel level to the semantic level through a progressive approach of "feature pyramid + cross-attention + graph topology matching". Especially in situations with large viewpoint differences and complex backgrounds, the consistency of the topological structure significantly reduces the misregistration rate and improves the registration success rate.

[0229] By introducing graph convolutional networks to encode the scene, feature points not only contain their own information but also structural context information from their neighborhood. This structure-aware capability makes the algorithm more stable when dealing with occlusion, changes in lighting, and viewpoint distortion. A fusion strategy based on gradient energy and structural similarity ensures that the output image retains the advantages of both infrared and visible light images while having sharper edges and a visual effect that better aligns with human visual perception, thus benefiting subsequent object detection and recognition tasks.

[0230] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0231] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for cross-view image registration and fusion based on pattern recognition, characterized in that, include: Collect multi-view images of the same scene, preprocess the multi-view images to obtain image pairs to be registered, and extract multi-scale feature maps from them to construct a feature pyramid; Based on the feature pyramid, cross-modal or cross-perspective feature interaction is performed using a cross-attention mechanism to generate an interactive feature map. Based on the interaction feature map, the initial spatial correspondence between the image pairs to be registered is predicted by a contrastive learning matching network, and the mismatches in the initial spatial correspondence are removed by a random sampling consensus algorithm to obtain the spatial correspondence. A scene semantic topology is constructed based on a graph structure. Cross-view feature matching relationships are established using the scene semantic topology. Geometric transformations are performed on the image pairs to be registered in combination with the spatial correspondence to obtain registered image pairs. Based on the gradient energy and structural similarity of the registered image pairs, a registration fusion optimization network is constructed, which takes the current registered image pair as input and outputs the fused image.

2. The pattern recognition based cross-view image registration and fusion method according to claim 1, characterized in that: The steps to obtain the image pair to be registered include: Determine the target scene and acquisition boundaries, select acquisition devices according to the requirements of the target scene, and acquire raw images according to the planned acquisition path, angle, and parameters; The original images are numbered according to the acquisition viewpoint order or modal type, and the acquisition viewpoint, time, and device parameters of each image are determined to obtain the multi-view image; The multi-view images are subjected to image specification unification, noise suppression, illumination and grayscale correction, and image smoothing enhancement to obtain preprocessed multi-view images; Based on the perspective association and modality type during acquisition, images with adjacent perspectives and feature overlap rates reaching a preset threshold are selected from the preprocessed multi-view images and paired to obtain the image pair to be registered.

3. The cross-view image registration and fusion method based on pattern recognition according to claim 2, characterized in that: The steps for constructing the feature pyramid include: The scale-invariant feature transform algorithm is used to extract corner points, edge features and corresponding feature descriptors from the image to be registered, and then mapped back to the original image to generate a first-scale feature map; The image to be registered is downsampled by half to obtain the sampled image scale. The mid-level features are extracted by combining the histogram of directional gradients and local binary mode, and mapped to the sampled image scale to obtain the second scale feature map. The sampled image scale is downsampled by half, and a lightweight convolutional neural network is used to extract deep semantic features. The deep semantic features are then deconvolutioned to generate a third-scale feature map. The first-scale feature map, the second-scale feature map, and the third-scale feature map are integrated in a preset order, and the initial pyramid is obtained by integrating them from top to bottom and aligning them horizontally. For each feature map of the initial pyramid, the Laplacian operator is used for edge enhancement, and the weights of each feature layer are dynamically adjusted according to the viewpoint differences of the image pair to be registered to obtain the feature pyramid.

4. The cross-view image registration and fusion method based on pattern recognition according to claim 1, characterized in that: The steps for generating the interaction feature map include: A strategy of scale-based fusion and bidirectional fusion is adopted to perform cross-attention feature interaction on each layer of the feature pyramid feature map; In each layer of feature maps, images A and B in the image pair to be registered are divided into feature blocks of a preset pixel size. The feature block of image A is used as the query and the feature block of image B is used as the key and value for bidirectional interaction. The similarity between feature blocks is calculated by performing a dot product operation between the query and the key, and then the cross-attention weights are obtained by normalizing them using the Softmax function. The cross-attention weights and their corresponding values ​​are weighted and summed to obtain the feature block-level feature map after bidirectional interaction. The feature maps corresponding to each layer of the feature pyramid are standardized in size, and then integrated using a fusion strategy of weighted summation and feature stitching to obtain the interactive feature map.

5. The cross-view image registration and fusion method based on pattern recognition according to claim 1, characterized in that: The steps to obtain the initial spatial correspondence include: The interactive feature map is projected and L2 normalized, and the feature vectors are distributed on the hypersphere to obtain feature image A and feature image B. The cosine similarity matrix between the two is calculated using matrix operations. For each feature point in the feature image A, the point with the highest score is found in the cosine similarity matrix, and the points in the feature image B are filtered by the mutual nearest neighbor filtering strategy to generate an initial set of matching point pairs. The initial set of matching point pairs is mapped back to the coordinates of the original image pairs to be registered by a downsampling factor, and then filtered according to a preset similarity threshold to obtain the initial spatial correspondence.

6. The cross-view image registration and fusion method based on pattern recognition according to claim 5, characterized in that: The steps to obtain the spatial correspondence include: The homography matrix model is determined to be a transformation model, and the hyperparameters of the transformation model are set according to cross-viewpoint and cross-modal scenarios; During the iteration process, multiple pairs of matching points are randomly selected from the initial spatial correspondence, and the transformation matrix of the transformation model is solved by direct linear transformation based on the coordinates of the matching points. All points in the feature image A are transformed to the coordinate system of the feature image B using the transformation matrix to obtain the predicted points; Calculate the distance between the predicted point and the actual matching point in the feature image B, and determine that points with a distance less than a preset distance threshold are internal points, and points with a distance greater than the preset distance threshold are external points; The number of interior points in the current model is counted, and it is determined whether it is greater than the number of interior points in the historical model. If so, the transformation model is updated. After the maximum number of iterations is reached, the set with the largest number of interior points is selected as the spatial correspondence.

7. The cross-view image registration and fusion method based on pattern recognition according to claim 5, characterized in that: The steps for establishing the cross-view feature matching relationship include: Semantic segmentation is performed on feature image A and feature image B to obtain the category label of each pixel. For each semantic region, edge detection is used to extract the contour. By analyzing connected components, pixels with the same category of labels, adjacent pixels, and connected pixels are merged into semantic target instances. The salient features of each semantic target instance are calculated to obtain the corresponding set of semantic targets. An undirected attribute graph structure is adopted to create nodes for each semantic target set. Based on preset requirements, it is determined whether to establish an edge and edge feature between any two nodes. Construct the scene semantic topology structure for feature image A and feature image B based on the nodes, edges, and edge features; The semantic topology of the scene is encoded using a graph convolutional network, and the feature information of each node and its neighboring nodes is aggregated to obtain the topology information node. The topological information nodes are embedded into the scene semantic topological structure, the morphological similarity of the topological information nodes of the feature image A and the feature image B is calculated, and a similarity matrix is ​​constructed by combining the comprehensive edge similarity after filtering. The node matching confidence is iteratively updated using a probabilistic relaxation method, and the node correspondence is generated by combining the similarity matrix. For each pair of matching nodes in the node correspondence, fine-grained feature matching is performed in the corresponding semantic region, and topological consistency filtering is performed to obtain the cross-view feature matching relationship.

8. The cross-view image registration and fusion method based on pattern recognition according to claim 1, characterized in that: The steps to obtain the registered image pair include: Based on the cross-perspective feature matching relationship and the spatial correspondence relationship, a constrained homography transformation model is constructed. A semantic topological structure regularization term is introduced, and weighted coefficients are assigned to the matching points according to the semantic node category. A weighted least squares optimization equation is constructed. An overdetermined linear system of equations is constructed using all matching nodes, and the geometric transformation matrix is ​​obtained by solving it in conjunction with topological constraints. The source image is inversely mapped according to the geometric transformation matrix, the corresponding coordinates of each pixel in the target image in the source image are calculated, and bilinear interpolation is used to complete pixel resampling to obtain the spatially aligned transformed image. The transformed image is subjected to local distortion correction to obtain a corrected transformed image. The image pair to be registered is combined with the corrected transformed image to obtain the registered image pair.

9. The cross-view image registration and fusion method based on pattern recognition according to claim 1, characterized in that: The steps for outputting the fused image include: The registered image pairs are normalized and denoised to extract the gradient energy and structural similarity. A basic architecture model is established using a four-stage structure of feature extraction, feature fusion, weight optimization, and image construction. The fusion loss function is constructed using the gradient energy, the structural similarity, and the pixel level. The basic architecture model is then enhanced and trained to obtain the registration and fusion optimization network. The processed registered image is input into the registration fusion optimization network. A preliminary fused image is obtained through network inference, and then optimized to obtain the final fused image.

10. A pattern recognition-based cross-view image registration and fusion system, applied to the pattern recognition-based cross-view image registration and fusion method as described in any one of claims 1 to 9, characterized in that, The system includes: The multi-scale feature construction module is used to acquire multi-view images of the same scene, preprocess the multi-view images to obtain image pairs to be registered, and extract multi-scale feature maps from them to construct a feature pyramid. The cross-domain interaction enhancement module is used to perform cross-modal or cross-perspective feature interaction based on the feature pyramid and utilize a cross-attention mechanism to generate an interaction feature map. The topology constraint matching module is used to predict the initial spatial correspondence between the image pairs to be registered based on the interaction feature map by using a contrastive learning matching network, and to remove mismatches in the initial spatial correspondence by using a random sampling consensus algorithm to obtain the spatial correspondence. The geometric alignment and registration module is used to construct a scene semantic topology based on a graph structure, establish a cross-view feature matching relationship using the scene semantic topology, and perform geometric transformation alignment on the image pair to be registered in combination with the spatial correspondence to obtain a registered image pair. The energy optimization fusion module is used to construct a registration fusion optimization network based on the gradient energy and structural similarity of the registered image pair, input the current registered image pair, and output the fused image.

Images