Fusion method for unmanned aerial vehicle cruise shot images and related equipment

Abstract

The embodiment of the invention discloses an unmanned aerial vehicle cruise shot image fusion method and related equipment, the local and global features of an infrared image and a visible light image are fully utilized, the extracted features are more representative and distinctive, matching is carried out based on the local features and the global features to obtain an affine transformation matrix, and the fusion efficiency is improved. According to the method, high-quality matching pairs can be quickly screened in an early stage, unnecessary calculation amount is reduced, resampling is performed on the infrared image by using the affine transformation matrix, the optimized infrared image is obtained, the optimized infrared image and the target visible light image are fused, wrong matching is effectively reduced, and the accuracy of feature matching is improved.

Description

[Technical field]

[0001] The present invention relates to the technical field of image fusion, and in particular to a method for fusing images shot by a cruise drone and related equipment. [Background technology]

[0002] With the widespread application of drone technology in the field of inspection, such as power inspection, industrial facility inspection, environmental monitoring, etc., simultaneous acquisition of infrared images and visible light images can provide richer information. However, due to factors such as changes in drone flight attitude and differences in shooting angles, it is difficult to accurately align these infrared images and visible light images from multiple angles in space, which affects the accurate analysis of inspection targets and subsequent work such as fault diagnosis. Traditional image registration methods cannot fully utilize the characteristics of infrared and visible light images, resulting in inaccurate feature extraction and low matching accuracy, which cannot meet the requirements for accurate target analysis during inspections. [Summary of the invention]

[0003] In view of this, the present invention provides a method for fusing images shot by a cruise drone and related equipment.

[0004] The specific technical scheme of the first embodiment of the present invention is: a method for fusing images taken by a drone during cruise, the method comprising: obtaining all infrared images and all visible light images taken by a drone during multi-angle cruise; extracting local color features for each infrared image and each visible light image, and obtaining a first local feature of each infrared image and a second local feature of each visible light image; the first local feature is a grayscale gradient directional histogram, and the second local feature is a color invariance feature; extracting global color features for each infrared image and each visible light image, and obtaining a first global feature of each infrared image and a second global feature of each visible light image; the global feature includes the total grayscale value of the image, the center of gravity of the image, and the direction and dispersion of the image; matching the first local feature, the second local feature, the first global feature, and the second global feature to obtain feature matching pairs of infrared images and visible light images; obtaining an affine transformation matrix based on the feature matching pairs; resampling the infrared image using the affine transformation matrix to obtain an optimized infrared image; fusing the optimized infrared image with a target visible light image; the coordinates of the target visible light image are the same as those of the optimized infrared image.

[0005] Preferably, the matching of the first local feature, the second local feature, the first global feature and the second global feature to obtain a feature matching pair of the infrared image and the visible light image includes: matching the first local feature with the second local feature, and matching the first global feature with the second global feature to obtain a matching similarity; and based on a value of the matching similarity, selecting a preset number of first local features and the second local features, or a first global feature and the second global feature, with high matching similarity as a feature matching pair.

[0006] Preferably, after selecting a preset number of first local features and the second local features, or a first global feature and the second global feature, with high matching similarity as feature matching pairs based on the numerical value of matching similarity, the method further includes: regarding the first feature points of the infrared image and the second feature points of the visible light image as nodes of a graph, regarding the similarity as the weight of the edge, and obtaining optimized feature matching pairs of the infrared image and the visible light image by solving the maximum weight matching problem of the graph; the first feature point includes a first local feature and a first global feature; the second feature point includes a second local feature and a second global feature; then obtaining an affine transformation matrix based on the feature matching pairs includes: obtaining an affine transformation matrix based on the optimized feature matching pairs.

[0007] Preferably, obtaining the affine transformation matrix according to the feature matching pairs includes: constructing an affine transformation model according to the feature matching pairs; and solving the affine transformation model by the least squares method to obtain the affine transformation matrix.

[0008] Preferably, after obtaining the affine transformation matrix, it also includes: minimizing the matching error of feature points as the objective function, iteratively optimizing the affine transformation matrix using a nonlinear optimization algorithm, and obtaining an optimized affine transformation matrix; then resampling the infrared image using the affine transformation matrix to obtain an optimized infrared image includes: resampling the infrared image using the optimized affine transformation matrix to obtain an optimized infrared image.

[0009] Preferably, after acquiring all the infrared images and all the visible light images taken by the drone during multi-angle cruising, the method further includes: preprocessing the infrared images and the visible light images to obtain optimized infrared images and optimized visible light images; the preprocessing includes removing noise and artifacts in the images; using the optimized infrared image as the infrared image, and using the optimized visible light image as the visible light image, returning to the step of extracting local color features from each infrared image and each visible light image to obtain the first local feature of each infrared image and the second local feature of each visible light image.

[0010] Preferably, the fusing of the optimized infrared image and the target visible light image includes: fusing the optimized infrared image and the target visible light image using a preset first weight coefficient and a preset second weight coefficient; the sum of the preset first weight coefficient and the preset second weight coefficient is 1.

[0011] The specific technical solution of the second embodiment of the present invention is: a fusion system for images taken by a drone during cruise, the system comprising: an image acquisition module, a local feature extraction module, a global feature extraction module, a feature matching module, a transformation matrix acquisition module, a resampling module and a fusion module; the image acquisition module is used to acquire all infrared images and all visible light images taken by the drone during multi-angle cruise; the local feature extraction module is used to perform color local feature extraction on each infrared image and each visible light image, and acquire the first local feature of each infrared image and the second local feature of each visible light image; the first local feature is a grayscale gradient directional histogram, and the second local feature is a color invariant feature; the global feature extraction module is used to perform color local feature extraction on each infrared image and each visible light image Color global feature extraction, obtaining the first global feature of each infrared image and the second global feature of each visible light image; the global features include the total grayscale value of the image, the center of gravity of the image, and the direction and dispersion of the image; the feature matching module is used to match the first local feature, the second local feature, the first global feature and the second global feature to obtain feature matching pairs of infrared images and visible light images; the transformation matrix acquisition module is used to obtain an affine transformation matrix based on the feature matching pairs; the resampling module is used to resample the infrared image using the affine transformation matrix to obtain an optimized infrared image; the fusion module is used to fuse the optimized infrared image with the target visible light image; the coordinates of the target visible light image are the same as those of the optimized infrared image.

[0012] The specific technical solution of the third embodiment of the present invention is: a fusion device for drone cruise shooting images, including a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method described in any one of the first embodiments of the present application.

[0013] The specific technical solution of the fourth embodiment of the present invention is: a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the method described in any one of the first embodiments of the present application.

[0014] Implementing the embodiments of the present invention will have the following beneficial effects:

[0015] The present invention makes full use of the local and global features of infrared images and visible light images to make the extracted features more representative and distinctive, and obtains an affine transformation matrix based on matching of local features and global features, which can quickly screen out high-quality matching pairs at an early stage and reduce unnecessary calculations. The infrared image is resampled using the affine transformation matrix to obtain an optimized infrared image, and the optimized infrared image is fused with the target visible light image, which effectively reduces mismatching and improves the accuracy of feature matching.

Brief Description of the Drawings

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 Flowchart of the steps of the fusion method for the images captured by the UAV cruise;

[0018] Figure 2 Schematic diagram of the structure of the fusion system for taking images for UAV cruising;

[0019] Figure 3 A diagram of the internal structure of a computer device;

[0020] Among them, 201, image acquisition module; 202, local feature extraction module; 203, global feature extraction module; 204, feature matching module; 205, transformation matrix acquisition module; 206, resampling module; 207, fusion module. [Specific implementation method]

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] The terms "first", "second", etc. in the specification, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other steps or modules that are inherent to these processes, methods, products or devices.

[0023] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0024] See also Figure 1 , is a flowchart of a method for fusing images shot by a drone cruise in the first embodiment of the present application, which improves the accuracy of feature matching. The method includes:

[0025] Step 101: Acquire all infrared images and all visible light images taken by the drone during multi-angle cruising;

[0026] Step 102: extracting local color features from each infrared image and each visible light image to obtain a first local feature of each infrared image and a second local feature of each visible light image; the first local feature is a grayscale gradient direction histogram, and the second local feature is a color invariance feature;

[0027] Step 103: extracting global color features from each infrared image and each visible light image to obtain a first global feature of each infrared image and a second global feature of each visible light image; the global features include a total grayscale value of the image, a center of gravity of the image, and a direction and dispersion of the image;

[0028] Step 104: Match the first local feature, the second local feature, the first global feature, and the second global feature to obtain a feature matching pair of the infrared image and the visible light image;

[0029] Step 105: Obtain an affine transformation matrix according to the feature matching pair;

[0030] Step 106: resample the infrared image using the affine transformation matrix to obtain an optimized infrared image;

[0031] Step 107: Fusing the optimized infrared image and the target visible light image; the target visible light image and the optimized infrared image have the same coordinates.

[0032] Specifically, the infrared image and visible light image are collected simultaneously by the infrared thermal imaging camera and visible light camera carried by the UAV; local features are extracted for the infrared image and visible light image respectively, wherein the local feature extraction adopts the improved scale-invariant feature transform (SIFT) algorithm, which is optimized based on the traditional SIFT algorithm for the characteristics of infrared image and visible light image. For infrared images, due to their low contrast and relatively unclear texture information, a feature enhancement step based on the grayscale gradient directional histogram is added to highlight the edge and contour features of the thermal target. For visible light images, color invariance feature description is added according to the color information of the image to improve the robustness of the feature to illumination changes; in addition to local features, global features of the image are extracted at the same time. The moment features of the image are calculated, including the zero-order moment (the total grayscale value of the image), the first-order moment (the center of gravity of the image) and the second-order moment (the direction and dispersion of the image) to obtain the overall shape and distribution information of the image. At the same time, the texture features of the image are extracted, such as the contrast, correlation, energy and entropy parameters of the grayscale co-occurrence matrix, to describe the image from the perspective of texture. The features of the extracted infrared image and visible light image are matched to obtain feature matching pairs of the infrared image and the visible light image; an affine transformation matrix is ​​constructed based on the feature matching pairs; and the infrared image is resampled using the affine transformation matrix to obtain an optimized infrared image; the optimized infrared image and the target visible light image are fused; specifically, the infrared image is resampled based on the optimized affine transformation matrix [Aopt]. For example, for a certain pixel point (x, y) in the infrared image, its corresponding coordinates (x'y') in the visible light image space are calculated by affine transformation, and the resampled pixel value is obtained by bilinear interpolation and other methods. Then, the infrared image and the visible light image are fused by the weighted average fusion method.

[0033] The local feature extraction is obtained by the following method: In the improved SIFT algorithm, for infrared images, the grayscale gradient directional histogram in the [8x8] neighborhood is calculated with each pixel as the center. For example, at the pixel with coordinates (x, y), the grayscale gradient [Gx, Gy] in the horizontal and vertical directions is calculated, the gradient direction θ is obtained by the inverse tangent function, and the gradient amplitude is assigned to the corresponding directional histogram bins. For visible light images, in addition to calculating the grayscale gradient, the color invariant feature descriptor is calculated according to its RGB color value. Assuming that the RGB values ​​at a certain pixel are [R, G, B], the color invariants [C1, C2, C3] are calculated by the following formula 1, and these color invariants are combined with the grayscale gradient information to construct a more robust local feature descriptor. After this step, [M1] local feature points are extracted in the infrared image and [M2] local feature points are extracted in the visible light image. Formula 1: Assume that the RGB values ​​of the pixel are R(x,y), G(x,y), and B(x,y), respectively, and calculate the color invariant using a specific formula, such as These color invariants are combined with the grayscale gradient directional histogram information to construct a more robust local feature descriptor vector FVIS. For example, color invariants can be added as an additional dimension to the feature descriptor vector, or integrated with grayscale gradient information through weighted fusion to improve the robustness of the feature to illumination changes.

[0034] The global feature extraction is obtained by the following method: Calculate the moment feature of the image. For infrared images, its zero-order moment [M00] is the sum of the gray values ​​of all pixels, and is obtained by the formula [M10 = ∑x∑yx l (x,y)] and [M01=∑xΣyy l (xy)] calculates the first-order moment, where [(x,y)] is the grayscale value of the pixel at the coordinate (x,y), and then obtains the centroid coordinates [Cx=M10 / M00, Cy=M01 / M00]. The calculation of the second-order moment involves the calculation of the covariance matrix, through which the direction and dispersion related parameters of the infrared image are obtained. For the visible light image, its moment characteristics are also calculated. At the same time, the grayscale co-occurrence matrix is ​​calculated. For example, the distance is set to [1] and the angle is [0°, 45°, 90°, 135°]. The grayscale co-occurrence matrix at different angles is calculated [GLCM0, GLCM45, GLCM 90, GLCM135], and then its contrast, correlation, energy, entropy and other texture feature parameters are calculated.

[0035] The method in this embodiment makes full use of the local and global features of the infrared image and the visible light image, making the extracted features more representative and distinctive, and obtains an affine transformation matrix based on matching of local features and global features, which can quickly screen out high-quality matching pairs at an early stage and reduce unnecessary calculations. The infrared image is resampled using the affine transformation matrix to obtain an optimized infrared image, and the optimized infrared image is fused with the target visible light image, which effectively reduces false matches and improves the accuracy of feature matching.

[0036] In a specific embodiment, the matching of the first local feature, the second local feature, the first global feature and the second global feature to obtain a feature matching pair of the infrared image and the visible light image includes: matching the first local feature with the second local feature, and matching the first global feature with the second global feature to obtain a matching similarity; and based on the value of the matching similarity, selecting a preset number of first local features and the second local features, or a first global feature and the second global feature, with high matching similarity as a feature matching pair.

[0037] Specifically, the similarity matrix [S] between the features of the infrared image and the visible light image is calculated from the local features and the global features, where the element [S(i,j)] represents the similarity between the [i]th feature point in the infrared image and the [j]th feature point in the visible light image. The weight of each matching pair is calculated based on the similarity [W(i,j)=f(S(i,j))], where f is a specific weight calculation function. [K] groups are randomly selected from all possible matching pairs, and the model (such as the affine transformation model) parameters are calculated after each selection, and the number of inliers (i.e., the number of matching pairs that satisfy the model and whose weights exceed a certain threshold) is calculated. After [R] iterations, the model with the largest number of inliers is selected as the initial model, and the initial matching pair set [P1] is obtained, which contains [P1_num] feature matching pairs.

[0038] In a specific embodiment, after selecting a preset number of first local features and the second local features, or a first global feature and the second global feature, with high matching similarity as a feature matching pair based on the matching similarity value, the method further includes: regarding the first feature point of the infrared image and the second feature point of the visible light image as nodes of a graph, regarding the similarity as the weight of the edge, and obtaining an optimized feature matching pair of the infrared image and the visible light image by solving the maximum weight matching problem of the graph; the first feature point includes a first local feature and a first global feature; the second feature point includes a second local feature and a second global feature; then obtaining an affine transformation matrix based on the feature matching pair includes: obtaining an affine transformation matrix based on the optimized feature matching pair.

[0039] Specifically, the feature points are regarded as nodes of the graph, and the similarity between the feature points is regarded as the weight of the edge to construct a bipartite graph. The Hungarian algorithm is used to solve the maximum weight matching problem of this bipartite graph. Assume that a new matching pair set [P2] is obtained by the Hungarian algorithm, which contains the matching pair [P2_num]. [P2_num] is usually greater than [P1_num] and has higher matching accuracy.

[0040] In a specific embodiment, obtaining the affine transformation matrix according to the feature matching pairs includes: constructing an affine transformation model according to the feature matching pairs; and solving the affine transformation model by the least squares method to obtain the affine transformation matrix.

[0041] Specifically, an affine transformation model is constructed based on the point pairs in the matching pair set [P2]. ​​Assume that the matching pairs are [(x1, y1) (points in the infrared image), (x2, y2) (points in the visible light image)], and the affine transformation model formula is:

[0042] x2=a 11 x1+a 12 y1+a 13

[0043] y2=a 21 x1+a 22 y1+a 23

[0044] Solve the above equations by the least squares method to obtain the affine transformation matrix A, that is:

[0045] [A=[a 11 ,a 12 ,a 13 ; a 21 ,a 22, a 23 ]].

[0046] In a specific embodiment, after obtaining the affine transformation matrix, it also includes: minimizing the matching error of feature points as the objective function, and iteratively optimizing the affine transformation matrix using a nonlinear optimization algorithm to obtain an optimized affine transformation matrix; then resampling the infrared image using the affine transformation matrix to obtain an optimized infrared image includes: resampling the infrared image using the optimized affine transformation matrix to obtain an optimized infrared image.

[0047] Specifically, the Levenberg-Marquardt algorithm is used to optimize the affine transformation model. Define the objective function [E = Σ(‖[x2,y2]-Ax[x1,y1]|) 2] (where ‖·‖ represents the Euclidean distance), and the objective function [E] is minimized by iteratively updating the affine transformation matrix [A]. Assume that after n iterations, the objective function value converges to [E_min], and the optimized affine transformation matrix [A_opt] is obtained.

[0048] In a specific embodiment, after acquiring all the infrared images and all the visible light images taken by the drone during multi-angle cruising, the method further includes: preprocessing the infrared images and the visible light images to obtain optimized infrared images and optimized visible light images; the preprocessing includes removing noise and artifacts in the images; using the optimized infrared image as the infrared image, and using the optimized visible light image as the visible light image, returning to the step of extracting local color features from each infrared image and each visible light image to obtain the first local feature of each infrared image and the second local feature of each visible light image.

[0049] Specifically, the formats of the collected infrared images and visible light images are standardized, the image resolution, color mode (for visible light images) and other parameters are unified, the noise and artifacts in the images are removed, and the image quality is improved.

[0050] In a specific embodiment, the fusing of the optimized infrared image and the target visible light image includes: fusing the optimized infrared image and the target visible light image using a preset first weight coefficient and a preset second weight coefficient; the sum of the preset first weight coefficient and the preset second weight coefficient is 1.

[0051] Specifically, suppose the fused image is [F(x,y)], the infrared image is [X(x;y)], the visible light image is [V(x;y)], and the fusion weights are [w1] and [w2] ([w1+w2=1]), then [F(x,y)=w1X|(x,y)+w2V(x,y)].

[0052] In a specific embodiment, [G] ground control points are pre-set on the power tower, and their coordinates in real space are known. The corresponding pixel coordinates of these ground control points are found in the registered image, and the error between them and the actual coordinates is calculated. For example, for the [i]th ground control point, its actual coordinate is [(X_real i, Yreal i)], and its coordinate in the registered image is [(X_image_i, Y_image_i)], and the error [Errori=√((X_real iX image_i) 2 +(Y_real i - Y_image i) 2)]. Calculate the average error of all ground control points [Average Error = ΣErrori / G]. If [Average_Error] is less than the preset threshold [Threshold], the registration is considered successful.

[0053] In the specific embodiments, see Figure 2 , is a structural schematic diagram of a fusion system for a drone cruise shooting image in the second embodiment of the present application, the system comprising: an image acquisition module 201, a local feature extraction module 202, a global feature extraction module 203, a feature matching module 204, a transformation matrix acquisition module 205, a resampling module 206 and a fusion module 207; the image acquisition module 201 is used to acquire all infrared images and all visible light images shot by the drone at multi-angle cruise; the local feature extraction module 202 is used to extract color local features for each infrared image and each visible light image, and acquire the first local feature of each infrared image and the second local feature of each visible light image; the first local feature is a grayscale gradient directional histogram, and the second local feature is a color invariant feature; the global feature extraction module 203 is used to extract the color local features for each infrared image and each visible light image A global color feature extraction is performed on each visible light image to obtain a first global feature of each infrared image and a second global feature of each visible light image; the global features include the total grayscale value of the image, the center of gravity of the image, and the direction and dispersion of the image; the feature matching module 204 is used to match the first local feature, the second local feature, the first global feature and the second global feature to obtain a feature matching pair of the infrared image and the visible light image; the transformation matrix acquisition module 205 is used to obtain an affine transformation matrix according to the feature matching pair; the resampling module 206 is used to resample the infrared image using the affine transformation matrix to obtain an optimized infrared image; the fusion module 207 is used to fuse the optimized infrared image with the target visible light image; the coordinates of the target visible light image are the same as those of the optimized infrared image.

[0054] The system in this embodiment makes full use of the local and global features of the infrared image and the visible light image, making the extracted features more representative and distinctive, and obtains an affine transformation matrix based on matching of local and global features, which can quickly screen out high-quality matching pairs at an early stage and reduce unnecessary calculations. The infrared image is resampled using the affine transformation matrix to obtain an optimized infrared image, and the optimized infrared image is fused with the target visible light image, effectively reducing false matches and improving the accuracy of feature matching.

[0055] In a specific embodiment, the third embodiment of the present application provides a fusion device for drone cruise shooting images, including a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods described in the first embodiment of the present application.

[0056] In a specific embodiment, the fourth embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method as described in any one of the first embodiments of the present application.

[0057] Figure 3 The internal structure of a computer device in one embodiment is shown. The computer device can be a terminal or a server. Figure 3 The computer device includes a processor, a memory, etc. connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement the method in this embodiment. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can implement the method in this embodiment. Those skilled in the art can understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0058] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

[0059] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for fusing images taken by a drone during cruise, characterized in that: The method comprises: Acquire all infrared images and all visible light images taken by the drone during multi-angle cruising; Performing color local feature extraction on each infrared image and each visible light image to obtain a first local feature of each infrared image and a second local feature of each visible light image; the first local feature is a grayscale gradient direction histogram, and the second local feature is a color invariance feature; Performing color global feature extraction on each infrared image and each visible light image to obtain a first global feature of each infrared image and a second global feature of each visible light image; the global features include a total grayscale value of the image, a center of gravity of the image, and a direction and dispersion degree of the image; Matching the first local feature, the second local feature, the first global feature, and the second global feature to obtain a feature matching pair of an infrared image and a visible light image; Obtaining an affine transformation matrix according to the feature matching pairs; Resampling the infrared image using the affine transformation matrix to obtain an optimized infrared image; The optimized infrared image and the target visible light image are fused; the coordinates of the target visible light image and the optimized infrared image are the same.

2. The method for fusing images taken by a drone during cruise according to claim 1, characterized in that: The step of matching the first local feature, the second local feature, the first global feature, and the second global feature to obtain a feature matching pair of the infrared image and the visible light image includes: Matching the first local feature with the second local feature, and matching the first global feature with the second global feature to obtain matching similarity; According to the value of the matching similarity, a preset number of first local features and the second local features, or a first global feature and the second global feature, with high matching similarity are selected as feature matching pairs.

3. The method for fusing images taken by a drone during cruise according to claim 2, characterized in that: After selecting a preset number of first local features and second local features with high matching similarity, or a first global feature and the second global feature with high matching similarity as feature matching pairs according to the value of the matching similarity, the method further includes: The first feature point of the infrared image and the second feature point of the visible light image are regarded as nodes of the graph, and the similarity is regarded as the weight of the edge. By solving the maximum weight matching problem of the graph, an optimized feature matching pair of the infrared image and the visible light image is obtained; the first feature point includes a first local feature and a first global feature; the second feature point includes a second local feature and a second global feature; Then obtaining an affine transformation matrix according to the feature matching pair includes: An affine transformation matrix is ​​obtained according to the optimized feature matching pairs.

4. The method for fusing images taken by a drone during cruise according to claim 1, characterized in that: The obtaining of an affine transformation matrix according to the feature matching pair comprises: constructing an affine transformation model according to the feature matching pairs; The affine transformation model is solved by the least square method to obtain the affine transformation matrix.

5. The method for fusing images taken by a drone during cruise as claimed in claim 4, characterized in that: After obtaining the affine transformation matrix, the method further includes: Taking the minimization of the matching error of the feature points as the objective function, and iteratively optimizing the affine transformation matrix using a nonlinear optimization algorithm to obtain an optimized affine transformation matrix; Then, the step of resampling the infrared image by using the affine transformation matrix to obtain an optimized infrared image includes: The infrared image is resampled using the optimized affine transformation matrix to obtain an optimized infrared image.

6. The method for fusing images taken by a drone during cruise according to claim 1, characterized in that: After obtaining all infrared images and all visible light images taken by the multi-angle cruise of the drone, the method further includes: Preprocessing the infrared image and the visible light image to obtain an optimized infrared image and an optimized visible light image; the preprocessing includes removing noise and artifacts in the image; The optimized infrared image is used as the infrared image, and the optimized visible light image is used as the visible light image, and the step of extracting local color features from each infrared image and each visible light image to obtain a first local feature of each infrared image and a second local feature of each visible light image is returned.

7. The method for fusing images taken by a drone during cruise according to claim 1, characterized in that: The step of fusing the optimized infrared image with the target visible light image comprises: The optimized infrared image and the target visible light image are fused using a preset first weight coefficient and a preset second weight coefficient; the sum of the preset first weight coefficient and the preset second weight coefficient is 1.

8. A fusion system for images taken by a drone during cruise, characterized in that: The system comprises: an image acquisition module, a local feature extraction module, a global feature extraction module, a feature matching module, a transformation matrix acquisition module, a resampling module and a fusion module; The image acquisition module is used to acquire all infrared images and all visible light images taken by the drone during multi-angle cruising; The local feature extraction module is used to extract color local features from each infrared image and each visible light image, and obtain a first local feature of each infrared image and a second local feature of each visible light image; the first local feature is a grayscale gradient direction histogram, and the second local feature is a color invariance feature; The global feature extraction module is used to extract color global features from each infrared image and each visible light image, and obtain a first global feature of each infrared image and a second global feature of each visible light image; the global features include the total gray value of the image, the center of gravity of the image, and the direction and dispersion of the image; The feature matching module is used to match the first local feature, the second local feature, the first global feature and the second global feature to obtain a feature matching pair of the infrared image and the visible light image; The transformation matrix acquisition module is used to obtain an affine transformation matrix according to the feature matching pair; The resampling module is used to resample the infrared image using the affine transformation matrix to obtain an optimized infrared image; The fusion module is used to fuse the optimized infrared image and the target visible light image; the coordinates of the target visible light image and the optimized infrared image are the same.

9. A device for fusing images taken by a drone during cruise, comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 7.

Images