Multi-source image automatic aligning and cutting method

Abstract

The invention discloses a multi-source image automatic aligning and cutting method, and belongs to the technical field of multi-source image processing. Comprising the steps of collecting a visible light image and an infrared image; extracting feature points of the visible light image and the infrared image; performing feature matching on the feature points of the visible light image and the infrared image to obtain multiple pairs of matching points; screening to obtain three pairs of matching points; performing affine transformation calculation to obtain an affine transformation matrix; calculating a public area of the visible light image and the infrared image; and mapping the public area to a coordinate system of the original visible light image by adopting an inverse matrix of an affine transformation matrix, thereby realizing cutting of the original visible light image. Through automatic feature point matching, affine transformation calculation and public area cutting, manual intervention is remarkably reduced, and the processing efficiency is improved. The method is suitable for a multi-source image registration scene, especially for the field of resource-limited unmanned aerial vehicle detection and medical image processing, and efficient and reliable image alignment and cutting are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-source image processing, and in particular relates to a multi-source image automatic alignment and cropping method. Background Art

[0002] In modern image processing, image alignment and registration are widely used in fields such as medical imaging, remote sensing monitoring, multispectral image processing, and computer vision. Different types of sensors and binocular camera systems used in these fields capture multiple images of the same scene. These images often differ in perspective, resolution, and geometry, requiring precise alignment through image alignment techniques to enable analysis and processing within the same coordinate system.

[0003] Existing image alignment methods mostly rely on geometric transformations and feature point matching techniques. However, these traditional methods have many shortcomings and limitations when dealing with multi-source heterogeneous data (such as visible light and infrared images captured by a binocular camera), as follows:

[0004] 1. Feature point matching results are unstable:

[0005] When processing multi-source images such as visible light and infrared, traditional feature point matching algorithms (such as SIFT and SURF) are susceptible to factors such as image perspective, lighting, and contrast. Feature point distribution can be uneven, leading to inaccurate matching point selection. For images with widely varying perspectives, overly dense or unevenly distributed matching points often occur, impacting subsequent alignment accuracy.

[0006] 2. Geometric transformation causes image distortion:

[0007] After matching feature points, traditional image alignment techniques typically perform an overall geometric transformation on the image through affine or perspective transformations. While these geometric transformations can align images, they can easily cause geometric distortion, especially when viewing angles vary significantly or the image content is complex. The transformed image may appear stretched, compressed, or tilted. This can destroy local image details, leading to a loss of geometric information from the original image and affecting the fidelity of the alignment results. This problem is particularly pronounced when processing multi-source images, such as remote sensing or medical images with varying resolutions and perspectives.

[0008] 3. Calculation of public areas is complex:

[0009] Because multi-source images typically come from different sensor devices (such as visible light and infrared cameras), their perspectives and resolutions vary, leading to differences in the image fields of view. Existing technologies often struggle to effectively calculate the common area between two images after image alignment. Many methods directly crop the aligned images, which can lose some valid information and easily retain invalid blank areas, reducing the effectiveness of image alignment and causing information loss and redundancy during precise analysis or fusion.

[0010] 4. Difficulty in automated processing:

[0011] Existing image alignment and cropping processes, especially for multi-source image registration, often require extensive manual intervention and adjustments. For example, the geometric transformation after feature point matching requires manual calibration, and the identification and cropping of common areas are difficult to fully automate. Without a sound cropping solution, existing methods struggle to achieve precise image alignment and cropping, potentially resulting in cumbersome and inefficient processing. Summary of the Invention

[0012] The purpose of the present invention is to address the above-mentioned deficiencies in the prior art and provide a method for automatic alignment and cropping of multi-source images to solve the problems of unstable feature point matching, image geometric deformation, and complex common area calculation in the existing multi-source image alignment technology.

[0013] In order to achieve the above object, the technical solution adopted by the present invention is:

[0014] A multi-source image automatic alignment and cropping method comprises the following steps:

[0015] S1, using binocular cameras to synchronously capture visible light images and infrared images in dynamic scenes;

[0016] S2, SuperPoint algorithm based on deep learning extracts feature points of visible light images and infrared images;

[0017] S3, using the SuperGlue algorithm to perform feature matching on the feature points of the visible light image and the infrared image to obtain multiple pairs of matching points;

[0018] S4, using the RANSAC algorithm to screen and obtain three pairs of matching points;

[0019] S5. Based on the three pairs of matching points, an affine transformation matrix is obtained by performing affine transformation calculation;

[0020] S6. Aligning the visible light image and the infrared image based on the affine transformation matrix, and calculating a common area between the visible light image and the infrared image;

[0021] S7. Use the inverse matrix of the affine transformation matrix to map the common area to the coordinate system of the original visible light image, thereby achieving cropping of the original visible light image.

[0022] Furthermore, in S2, feature points of the visible light image and the infrared image are extracted, including:

[0023]

[0024] Where I is the input image, including visible light image and infrared image; θ p and θ d are the parameters of the feature point detector and descriptor extractor respectively; is the set of detected feature points, is the corresponding feature descriptor set; f() represents the process of extracting the feature point set through the feature point detector; g() represents the process of generating descriptors for the feature points through the descriptor extractor.

[0025] Furthermore, S3 includes the following sub-steps:

[0026] S31. Use Euclidean distance to calculate the familiarity between two feature points:

[0027] d(p1,p2)=||D1-D2||

[0028] Where d(p1,p2) is the distance between feature point p1 and feature point p2; D1 and D2 are the feature descriptor vectors of the corresponding feature points respectively; |||| is the Euclidean norm;

[0029] S32. Based on the familiarity between the two feature points, a graph neural network is used to transfer feature point information to obtain an optimized feature point.

[0030] S33. Calculate similarity scores of any two feature points in the visible light image and the infrared image based on the optimized feature points, and use the Sinkhorn algorithm to complete feature point matching between the visible light image and the infrared image based on the similarity scores to obtain several pairs of matching points.

[0031] Furthermore, in S32, a graph neural network is used to transmit feature point information, including:

[0032]

[0033] Where, is the feature representation of the (t+1)th layer feature point after feature point i passes the message; w ij is the weight between feature points i and j obtained based on the attention mechanism; is the neighborhood of feature point i, and the points in this area are determined based on the familiarity calculation in S31; is the feature representation of the t-th layer feature point after feature point j passes the message;

[0034] Among them, the attention mechanism is used to dynamically calculate the weight w between feature points i and j ij :

[0035]

[0036] Where, is the similarity between feature point i and feature point j.

[0037] Furthermore, S33 specifically includes:

[0038] Calculate the similarity score of any two feature points in the visible light image and the infrared image:

[0039]

[0040] Where S ij is the similarity score between feature point i and feature point j; is the optimized feature representation of feature point i in the visible light image after the feature point information is transmitted through the graph neural network. is the optimized feature representation of feature point j in the infrared image after the feature point information is transmitted through the graph neural network;

[0041] Based on multiple similarity scores S ij Get the similarity score matrix S; use the Sinkhorn algorithm to regularize the similarity matrix S and convert the similarity matrix S into a matching probability matrix The matching probability matrix The matching probability in is compared with the matching probability threshold, and the feature point pairs are screened to obtain several pairs of matching points.

[0042] Furthermore, S4 specifically includes:

[0043] A random sampling consensus algorithm is used to screen all matching points. Among the screened matching points, three pairs of matching points with the smallest distance d(p1, p2) are selected:

[0044] mind(p1,p2)=||D1-D2||

[0045] The three smallest pairs of matching points obtained by screening satisfy:

[0046] d(p i ,p j )>∈i≠j

[0047] Where ∈ is the distance threshold.

[0048] Furthermore, in S5, an affine transformation is calculated to obtain an affine transformation matrix, including:

[0049]

[0050] Where A is the affine transformation matrix; (x, y) is the coordinate point in the original visible light image, (x′, y′) is the coordinate of the corresponding point (x, y) in the infrared image; a 11 , a 12 , a 21 , a 22 is the transformation parameter; translation vector t=(t x ,t y ), t x ,t y are the horizontal and vertical translations of the image, respectively.

[0051] Furthermore, S6 includes:

[0052] Based on the affine transformation matrix, the visible light image is transformed into the coordinate system of the infrared image:

[0053] I v′ =cv2.warpAffine(I v ,A,(w,h))

[0054] Where, I' v is the visible light image after affine transformation; cv2.warpAffine is the function used to perform affine transformation in OpenCV; I v is the visible light image; w, h are the width and height of the infrared image;

[0055] Use logical operations to calculate the common area between the visible light image and the infrared image:

[0056] B common =B v′ ∧B ir

[0057] Among them: B v′ =cv2.threshold(I v′ ,T,255,cv2.THRESH B INARY)

[0058] B ir =cv2.threshold(I ir ,1,255,cv2.THRESH B INARY)

[0059] Where B commonis the intersection of the binary images of the visible light image and the infrared image, indicating the overlapping part between the visible light image and the infrared image; ∧ is the logical AND operation; Bv′ is the transformed image after binarization, B ir is the binarized version of the infrared image; cv2.threshold is the image binarization function in OpenCV; T is the threshold; cv2.THRESH B INARY is a binary mode, where pixel values greater than the threshold are set to white, and those below the threshold are set to black;

[0060] According to the intersection B of the binary images of the visible light image and the infrared image common , calculate the bounding box of the common area:

[0061] (x v ,y v ,w common ,h common )=cv2.boundingRect(B common )

[0062] In the formula, (x v ,y v ) are the vertex coordinates of the common area, where w common ,h common is the width and height of the public area; x v ,y v are the coordinates of the upper left corner of the common area respectively; cv2.boundingRect is a function in OpenCV used to calculate the minimum bounding rectangle of the image contour.

[0063] Furthermore, S7 includes:

[0064] Map the common area to the coordinate system of the original visible light image:

[0065]

[0066] Where, are the coordinates of the common area in the transformed image; is the coordinate of the common area in the original visible light image after inverse transformation; t -1 is the inverse of the translation vector.

[0067] Furthermore, the inverse matrix A of the affine transformation matrix -1 for:

[0068]

[0069] The multi-source image automatic alignment and cropping method provided by the present invention has the following beneficial effects:

[0070] 1. The present invention uses an advanced feature point detection algorithm to automatically extract and screen three pairs of matching points with high confidence and reasonable distribution in multi-source images. In this way, high-precision matching can be achieved between multi-source images with large differences in viewing angles and resolutions, significantly improving the accuracy and robustness of image registration.

[0071] 2. The present invention does not directly perform an affine transformation on the entire image, but instead calculates the common area of the two images based on the affine transformation. This method effectively avoids the distortion problem commonly encountered in the geometric adjustment process, thereby maintaining the natural proportions and geometric information of the image. It is particularly suitable for dynamic scenes in drone monitoring and scenes such as medical imaging with high precision requirements.

[0072] 3. This method calculates the common area and employs cropping techniques to ensure that only overlapping portions of the image are retained, avoiding invalid areas. Image binarization and logical operations are used to accurately determine the common area, and bounding box calculations are used to determine the cropping boundaries. This method preserves the natural proportions and detail integrity of the image, improving visual consistency and information accuracy after image alignment.

[0073] 4. This method significantly reduces manual intervention and improves processing efficiency through automated feature point matching, affine transformation calculation, and common area cropping. This method is suitable for multi-source image registration scenarios, particularly in resource-constrained areas such as drone detection and medical image processing, achieving efficient and reliable image alignment and cropping.

[0074] 5. This invention avoids the need for a direct global affine transformation through an innovative approach that combines precise feature point matching with common area cropping. Traditionally, affine transformations often result in geometric distortions of the image, such as stretching, compression, or skewing. However, by calculating the common area of the image and performing cropping on the original image, this invention ensures the image's natural proportions and geometric integrity, avoiding image distortion caused by direct geometric transformations. This is particularly important for applications requiring high-precision image analysis, such as fire detection in drones or bird monitoring.

[0075] 6. The present invention utilizes the SuperPoint and SuperGlue feature point matching algorithms, combined with RANSAC (random sampling consensus algorithm) to screen the matching points to ensure that the matching points have high confidence and accuracy. RANSAC effectively eliminates outliers in the matching points, further improving the stability and accuracy of the affine transformation. After the matching is completed, the present invention ensures the accuracy of the affine transformation calculation through an innovative three-pair optimal matching point screening strategy. In order to avoid the three pairs of matching points being too concentrated in the local area of the image, the Euclidean distance between the matching points is calculated, and a distance threshold is set to ensure that the spatial distribution of the three pairs of matching points is reasonable. This innovative method is suitable for complex scenes such as drone fire monitoring and bird recognition, effectively improving the accuracy of image alignment and reducing the false detection rate caused by local deformation.

[0076] 7. This invention achieves efficient resource utilization by cropping common regions rather than direct geometric transformations. Traditional image alignment methods typically require transforming and processing the entire image, resulting in a waste of computing resources and network bandwidth. This invention, however, crops only the common regions of the image, significantly reducing the amount of data processed and thus saving computing resources and storage space. This makes the invention particularly suitable for resource-constrained environments such as drones, enabling faster responses and significantly improving processing efficiency in tasks such as fire monitoring and bird identification.

[0077] 8. The present invention's inverse affine transformation and common area cropping techniques enable the present invention to maintain high precision while avoiding the processing of invalid areas. By mapping the common area back to the coordinate system of the original image and cropping it, the present invention ensures the effectiveness of the aligned image content, eliminates invalid boundaries and blank areas, and thus further enhances the visual consistency and information accuracy of the image. This can effectively improve the system's recognition accuracy of key targets (such as flames, smoke, flying birds, etc.) in low-altitude drone monitoring missions and reduce the false detection rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a flow chart of a method for automatic alignment and cropping of multi-source images according to the present invention. DETAILED DESCRIPTION

[0079] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0080] Example 1

[0081] The multi-source image automatic alignment and cropping method of this embodiment can be used to solve the problems of geometric deformation and information loss in the alignment of multi-source heterogeneous images when drones monitor dynamic scenes such as fires and flying birds. Through this method, images can be accurately aligned while maintaining natural proportions and details, ensuring the reliability of analysis and processing results. Figure 1 , which specifically includes the following steps:

[0082] Step S1: using a binocular camera to synchronously capture visible light images and infrared images of a dynamic scene;

[0083] Specifically, this embodiment can be applied to drones monitoring dynamic scenes such as fires and flying birds, using a binocular camera to synchronously capture visible light images and infrared images. Due to the complex acquisition environment, such as lighting changes, obstacle obstructions, signal noise and other factors, some images may have poor quality (such as blurred or partially obscured images). Therefore, it is necessary to effectively screen the images and delete invalid images to ensure the stability and accuracy of subsequent processing.

[0084] Step S2: Extracting feature points of visible light images and infrared images using the SuperPoint algorithm based on deep learning, which specifically includes the following:

[0085] This embodiment adopts the SuperPoint algorithm based on deep learning as the core algorithm for feature advance. SuperPoint is a feature point detection and descriptor extraction algorithm based on convolutional neural network (CNN). It uses synthetic data sets for pre-training and learns feature point detection through image reconstruction tasks in the absence of labeled data. It solves the problem that traditional feature point detection algorithms (such as SIFT, SURF, ORB, etc.) have poor detection effects in complex scenes or images of different modalities (such as visible light and infrared). SuperPoint first uses a convolutional neural network (CNN) to detect key points from the input image. By performing a convolution operation on the image, a feature map is generated, and then a non-maximum suppression (NMS) operation is applied to each feature map to extract sparse key points in the image. After detecting the key points, SuperPoint uses another convolutional neural network to extract local image feature descriptors for each key point. The specific extraction process is as follows:

[0086]

[0087] Where I is the input image, including visible light image and infrared image; θ p and θ d are the parameters of the feature point detector and descriptor extractor respectively; is the set of detected feature points, is the corresponding feature descriptor set; f() represents the process of extracting a feature point set through a feature point detector, and the output is the spatial coordinates or image position of the feature point; g() represents the process of generating descriptors for feature points through a descriptor extractor, and the output is a high-dimensional feature vector that corresponds one-to-one to the feature point position.

[0088] Step S3: Use the SuperGlue algorithm to perform feature matching on the feature points of the visible light image and the infrared image to obtain multiple pairs of matching points. The core of SuperGlue is to perform global association modeling on the feature points in the two images by building a graph neural network. Combined with the attention mechanism, it learns the matching relationship between the feature points of the two images. It specifically includes the following:

[0089] Step S31: Calculate the familiarity between two feature points using Euclidean distance:

[0090] d(p1,p2)=||D1-D2||

[0091] Where d(p1, p2) is the distance between feature point p1 and feature point p2; D1 and D2 are the feature descriptor vectors of the corresponding feature points respectively; |||| is the Euclidean norm, which is used to calculate the distance between vectors. The smaller the distance, the more similar the descriptors of the feature points are, and the higher the possibility of matching.

[0092] Euclidean distance is used as the initial similarity measure to help establish the initial connection between the feature points in the two images, that is, to construct the initial adjacency relationship graph based on the Euclidean distance, where the neighborhood Contains the feature point p i A point closer.

[0093] Step S32: Based on the familiarity between the two feature points, a graph neural network is used to transfer feature point information to obtain optimized feature points;

[0094] Specifically, SuperGlue introduces a graph neural network based on this, optimizing the similarity calculation of feature points through message passing and attention mechanisms, thereby achieving more robust matching. SuperGlue's graph neural network is based on information transmission between nodes (feature points), including:

[0095]

[0096] Where, is the feature representation of the (t+1)th layer feature point after feature point i passes the message; w ij is the weight between feature points i and j obtained based on the attention mechanism; is the neighborhood of feature point i, and the points in this area are determined based on the familiarity calculation in S31; is the feature representation of the t-th layer feature point after feature point j passes the message;

[0097] Through message passing, graph neural networks can dynamically aggregate information from neighboring nodes, so that the representation of each feature point gradually integrates global context information.

[0098] Among them, the attention mechanism is used to dynamically calculate the weight w between feature points i and j ij :

[0099]

[0100] Where, It is the similarity between feature point i and feature point j. Its function is to dynamically adjust the intensity of information transmission so that the feature point can pay more attention to the neighboring nodes with higher matching correlation with itself, thereby improving the accuracy and robustness of the feature point similarity calculation.

[0101] Finally, through multi-layer message passing, the representation of feature points is gradually optimized to a global context-aware feature representation.

[0102] Step S33: Calculate the similarity scores of any two feature points in the visible light image and the infrared image based on the optimized feature points, and use the Sinkhorn algorithm to match the feature points between the visible light image and the infrared image based on the similarity scores to obtain several pairs of matching points:

[0103] Calculate the similarity score of any two feature points in the visible light image and the infrared image:

[0104]

[0105] Where S ij is the similarity score between feature point i and feature point j; is the optimized feature representation of feature point i in the visible light image after the feature point information is transmitted through the graph neural network. is the optimized feature representation of feature point j in the infrared image after the feature point information is transmitted through the graph neural network;

[0106] This embodiment is based on multiple similarity scores S ij Get the similarity score matrix S; use the Sinkhorn algorithm to regularize the similarity matrix S and convert the similarity matrix S into a matching probability matrix This ensures that the matching between feature points is unique, that is, each feature point in one image can only be matched to one feature point in another image; the matching probability matrix The matching probability in is compared with the matching probability threshold to screen out possible feature point pairs, and then obtain several pairs of matching points.

[0107] Step S4: Use the RANSAC algorithm to screen and obtain three pairs of matching points;

[0108] Specifically, a two-step screening mechanism is introduced, combined with the RANSAC algorithm, to ensure that the three pairs of matching points finally selected have high confidence and reasonable spatial distribution, thereby ensuring the accuracy of the affine transformation;

[0109] The first step is screening, selecting high confidence matching points;

[0110] The present invention uses RANSAC (Random Sample Consensus Algorithm) to screen all matching points. RANSAC randomly selects three pairs of matching points to calculate the affine transformation matrix and removes outliers that do not conform to the global geometric relationship. This ensures that only high-confidence inliers that conform to the global geometric relationship are retained, improving matching accuracy.

[0111] After filtering out the inliers, the present invention further selects three pairs of points with the smallest matching distance d(p1, p2) as the final affine transformation basis, that is, among the filtered matching points, select three pairs of matching points with the smallest distance d(p1, p2):

[0112] mind(p1,p2)=||D1-D2||

[0113] The second step is screening, selecting matching points with reasonable distribution;

[0114] To avoid the three pairs of matching points from being too concentrated in a local area of the image and to ensure their reasonable spatial distribution, the present invention calculates the Euclidean distance between the matching points and sets a distance threshold ∈. During the screening process, the distance d(p1, p2) between each pair of matching points is required to satisfy:

[0115] d(p i ,p j )>∈i≠j

[0116] Where ∈ is the distance threshold.

[0117] The screening constraints in this embodiment ensure that the three selected matching point pairs are evenly distributed, avoiding excessive concentration in a small area of the image. This distribution constraint can reduce alignment failures caused by local deformation or occlusion, thereby improving the robustness and accuracy of image registration.

[0118] Step S5: Based on the three pairs of matching points, an affine transformation matrix is obtained by performing affine transformation calculation;

[0119] Specifically, affine transformation can describe the linear transformation relationship of an image during rotation, scaling, translation, and tilt. An important feature of affine transformation is that the matrix can be uniquely determined by the coordinates of three pairs of matching points. The affine transformation matrix is calculated using affine transformation, including:

[0120]

[0121] Where A is the affine transformation matrix; (x, y) is the coordinate point in the original visible light image, (x′, y′) is the coordinate of the corresponding point (x, y) in the infrared image; a 11 , a 12 , a 21 , a 22 is a transformation parameter used to describe transformations such as rotation, scaling, and tilt; translation vector t=(t x ,t y ), t x ,t y are the horizontal and vertical translations of the image, respectively.

[0122] Step S6: aligning the visible light image and the infrared image based on the affine transformation matrix, and calculating the common area of the visible light image and the infrared image;

[0123] Specifically, the matrix A obtained in step S5 represents geometric transformations such as rotation, scaling, and tilting, and the translation vector t is responsible for moving the image. However, if it is directly used to perform an overall affine transformation to process the alignment of visible light images and infrared images, it will often cause image disproportion, and even produce distortion effects such as stretching and compression. Based on this, this embodiment adopts a new approach to solve the problem of multi-source image alignment: after preliminarily aligning the images through affine transformation, the images are not directly transformed as a whole, but the common areas of the two images are calculated, and only the valid parts of their overlap are retained. This strategy can avoid the distortion caused by the overall transformation and retain the natural proportions and details of the image;

[0124] Based on the affine transformation matrix, the visible light image is transformed into the coordinate system of the infrared image:

[0125] I v′ =cv2.warpAffine(I v ,A,(w,h))

[0126] Where, I' v is the visible light image after affine transformation; cv2.warpAffine is the function used to perform affine transformation in OpenCV, which is to transform the input image I v Map the pixel positions according to the specified affine transformation matrix A to generate the transformed output image I' v;I v is the visible light image; w, h are the width and height of the infrared image;

[0127] Then, in order to calculate the common area of the two images, this embodiment uses image binarization to convert the pixel values of the images into 0 and 1. The binarization operation in image processing can be represented by a threshold function:

[0128]

[0129] In OpenCV, binarization is achieved through cv2.threshold:

[0130] B v′ =cv2.threshold(I v′ ,T,255,cv2.THRESH B INARY)

[0131] B ir =cv2.threshold(I ir ,1,255,cv2.THRESH B INARY)

[0132] Next, logical operations are used to calculate the common area of the two images, and bitwise_and operations are used to calculate the overlapping area between binary images; that is, logical operations are used to calculate the common area between the visible light image and the infrared image:

[0133] B common =B v′ ∧B ir

[0134] Where B common is the intersection of the binary images of the visible light image and the infrared image, which represents the overlapping part between the visible light image and the infrared image; ∧ is the logical AND operation that can be used to extract the overlapping part of the two images. v′ and B ir The result is 1 (white) only when the corresponding pixel values are all 1 (white). Otherwise, the result is 0 (black); Bv′ is the transformed image after binarization, B ir is the binarized version of the infrared image; cv2.threshold is a commonly used image binarization function in OpenCV. It divides the pixel values in the image into two categories: pixels below the threshold are set to 0 (black), and pixels above the threshold are set to 255 (white); T is the threshold, which is the dividing line that controls the binarization. For visible light images, a dynamic threshold is used, and for infrared images, the threshold is fixed at 1; cv2.THRESH BINARY is a binary mode, where pixel values greater than the threshold are set to white and those below the threshold are set to black. This ensures that the calculated common area only contains valid information from the two images and excludes invalid background areas.

[0135] In order to further determine the specific range of the common area, the present invention calculates the bounding box of the common area through the cv2.boundingRect() function:

[0136] (x v ,y v ,w common ,h common )=cv2.boundingRect(B common )

[0137] In the formula, (x v ,y v ) are the vertex coordinates of the common area, where w common ,h common is the width and height of the public area; x v ,y v are the upper left corner coordinates of the common area respectively; cv2.boundingRect is a function in OpenCV used to calculate the minimum bounding rectangle of the image outline. This method returns the upper left corner coordinates of the common area (x v ,y v ) and the width w and height h of the common area. Through this bounding box, the present invention can accurately determine the common area of the two images.

[0138] Step S7: Mapping the common area to the coordinate system of the original visible light image using the inverse matrix of the affine transformation matrix, thereby cropping the original visible light image. This avoids directly operating the image after the affine transformation, ensuring the geometric naturalness and true proportion of the image;

[0139] Among them, the inverse matrix of the affine transformation matrix is:

[0140]

[0141] In this embodiment, the inverse affine transformation matrix A can be calculated by using the cv2.invertAffineTransform() function provided by OpenCV. -1 , this matrix maps the common area from the transformed image coordinate system back to the coordinate system of the original visible light image.

[0142] The detection results of the common area are usually given as a rectangular area with vertex coordinates {x, y, w, h}, which represent the upper left corner (x, y) of the common area, as well as the width w and height h. To map this area back to the coordinate system of the original visible light image, the coordinates of these vertices are adjusted by combining the inverse affine transformation.

[0143] Map the common area to the coordinate system of the original visible light image:

[0144]

[0145] Where, are the coordinates of the common area in the transformed image; is the coordinate of the common area in the original image after inverse transformation; t -1 is the inverse of the translation vector.

[0146] After the cropping operation is completed, the final output image contains the common areas of the visible light image and the infrared image, and maintains the geometric information and proportions of the original image. This embodiment is fully automated and is suitable for multi-source image processing scenarios in dynamic drone scenes, such as fire monitoring and bird detection.

[0147] Although the specific embodiments of the invention are described in detail in conjunction with the accompanying drawings, this should not be construed as limiting the scope of protection of this patent. Within the scope described by the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of this patent.

Claims

1. A method for automatic alignment and cropping of multi-source images, characterized in that: The following steps are involved: S1, using binocular cameras to synchronously capture visible light images and infrared images in dynamic scenes; S2, SuperPoint algorithm based on deep learning extracts feature points of visible light images and infrared images; S3, using the SuperGlue algorithm to perform feature matching on the feature points of the visible light image and the infrared image to obtain multiple pairs of matching points; S4, using the RANSAC algorithm to screen and obtain three pairs of matching points; S5. Based on the three pairs of matching points, an affine transformation matrix is obtained by performing affine transformation calculation; S6. Aligning the visible light image and the infrared image based on the affine transformation matrix, and calculating a common area between the visible light image and the infrared image; S7. Use the inverse matrix of the affine transformation matrix to map the common area to the coordinate system of the original visible light image, thereby achieving cropping of the original visible light image.

2. The multi-source image automatic alignment and cropping method according to claim 1, characterized in that: In S2, extracting feature points of the visible light image and the infrared image includes: Where I is the input image, including visible light image and infrared image; θ p and θ d are the parameters of the feature point detector and descriptor extractor respectively; is the set of detected feature points, is the corresponding feature descriptor set; f() represents the process of extracting the feature point set through the feature point detector; g() represents the process of generating descriptors for the feature points through the descriptor extractor.

3. The multi-source image automatic alignment and cropping method according to claim 1, characterized in that: The S3 includes the following sub-steps: S31. Use Euclidean distance to calculate the familiarity between two feature points: d(p1,p2)=||D1-D2|| Where d(p1,p2) is the distance between feature point p1 and feature point p2; D1 and D2 are the feature descriptor vectors of the corresponding feature points respectively; |||| is the Euclidean norm; S32. Based on the familiarity between the two feature points, a graph neural network is used to transfer feature point information to obtain an optimized feature point. S33. Calculate similarity scores of any two feature points in the visible light image and the infrared image based on the optimized feature points, and use the Sinkhorn algorithm to complete feature point matching between the visible light image and the infrared image based on the similarity scores to obtain several pairs of matching points.

4. The multi-source image automatic alignment and cropping method according to claim 3, characterized in that: In S32, the feature point information is transmitted using a graph neural network, including: Where, f i (t+1) is the feature representation of the (t+1)th layer feature point after feature point i passes the message; w ij is the weight between feature points i and j obtained based on the attention mechanism; is the neighborhood of feature point i, and the points in this area are determined based on the familiarity calculation in S31; is the feature representation of the t-th layer feature point after feature point j passes the message; Among them, the attention mechanism is used to dynamically calculate the weight w between feature points i and j ij : Where, is the similarity between feature point i and feature point j.

5. The multi-source image automatic alignment and cropping method according to claim 3, characterized in that: The S33 specifically includes: Calculate the similarity score of any two feature points in the visible light image and the infrared image: Where S ij is the similarity score between feature point i and feature point j; f i (L) is the optimized feature representation of feature point i in the visible light image after the feature point information is transmitted through the graph neural network. is the optimized feature representation of feature point j in the infrared image after the feature point information is transmitted through the graph neural network; Based on multiple similarity scores S ij Get the similarity score matrix S; use the Sinkhorn algorithm to regularize the similarity matrix S and convert the similarity matrix S into a matching probability matrix The matching probability matrix The matching probability in is compared with the matching probability threshold, and the feature point pairs are screened to obtain several pairs of matching points.

6. The multi-source image automatic alignment and cropping method according to claim 3, characterized in that: The S4 specifically includes: A random sampling consensus algorithm is used to screen all matching points. Among the screened matching points, three pairs of matching points with the smallest distance d(p1, p2) are selected: mind(p1,p2)=||D1-D2|| The three smallest pairs of matching points obtained by screening satisfy: d(p i ,p j )>∈i≠j Where ∈ is the distance threshold.

7. The multi-source image automatic alignment and cropping method according to claim 1, characterized in that: In S5, an affine transformation is performed to obtain an affine transformation matrix, including: Where A is the affine transformation matrix; (x, y) is the coordinate point in the original visible light image, (x′, y′) is the coordinate of the corresponding point (x, y) in the infrared image; a 11 , a 12 , a 21 , a 22 is the transformation parameter; translation vector t=(t x ,t y ), t x ,t y are the horizontal and vertical translations of the image, respectively.

8. The multi-source image automatic alignment and cropping method according to claim 7, characterized in that: The S6 includes: Based on the affine transformation matrix, the visible light image is transformed into the coordinate system of the infrared image: I v′ =cv2.warpAffine(I v ,A,(w,h)) Where, I' v is the visible light image after affine transformation; cv2.warpAffine is the function used to perform affine transformation in OpenCV; I v is the visible light image; w, h are the width and height of the infrared image; Use logical operations to calculate the common area between the visible light image and the infrared image: B common =B v′ ∧B ir Where: B v′ = cv2.threshold(I v′ , T, 255, cv2.THRESH B INARY) B ir =cv2.threshold(I ir ,1,255,cv2.THRESH B INARY) Where B common is the intersection of the binary images of the visible light image and the infrared image, indicating the overlapping part between the visible light image and the infrared image; ∧ is the logical AND operation; Bv′ is the transformed image after binarization, B ir is the binarized version of the infrared image; cv2.threshold is the image binarization function in OpenCV; T is the threshold; cv2.THRESH B INARY is a binary mode, where pixel values greater than the threshold are set to white, and those below the threshold are set to black; According to the intersection B of the binary images of the visible light image and the infrared image common , calculate the bounding box of the common area: (x v ,y v ,w common ,h common )=cv2.boundingRect(B common ) In the formula, (x v ,y v ) are the vertex coordinates of the common area, where w common ,h common is the width and height of the public area; x v ,y v are the coordinates of the upper left corner of the common area respectively; cv2.boundingRect is a function in OpenCV used to calculate the minimum bounding rectangle of the image contour.

9. The multi-source image automatic alignment and cropping method according to claim 7, characterized in that: The S7 includes: Map the common area to the coordinate system of the original visible light image: Where, are the coordinates of the common area in the transformed image; is the coordinate of the common area in the original visible light image after inverse transformation; t -1 is the inverse of the translation vector.

10. The multi-source image automatic alignment and cropping method according to claim 9, characterized in that: The inverse matrix A of the affine transformation matrix -1 for:

Images