Zoom mismatch adjustment method for heterogeneous image fusion based on edge gradient mutual information

Abstract

The present invention discloses a method for adjusting zoom mismatch of heterogeneous image fusion based on edge gradient mutual information, which belongs to the field of heterogeneous image fusion. The present invention extracts edges from pre-processed heterogeneous input images, performs normalized mutual information statistics on heterogeneous edge grayscale images, and then performs gradient weighted statistics on edge images, and uses it as the weight of edge normalized mutual information, and uses edge gradient mutual information as the evaluation index of heterogeneous field of view matching; adopts a hill climbing search method improved by optimizing the search direction change strategy, uses three adjacent images as criteria, searches for the maximum value of the evaluation index, that is, the most matching position, and solves the problem of slight mismatch of field of view of continuous zoom heterogeneous fusion imaging system. The present invention improves the grayscale correlation of heterogeneous images, constructs gradient weighting that is sensitive to slight changes in field of view, and enhances the unimodality and sensitivity of the evaluation index; the present invention can effectively suppress the influence of local extreme values ​​and improve zoom registration accuracy.

Description

Technical Field

[0001] The invention relates to a heterogeneous image fusion zoom mismatch adjustment method based on edge gradient mutual information, and belongs to the field of heterogeneous image fusion. Background Art

[0002] In recent years, continuous zoom imaging detector technology has been applied in continuous zoom heterogeneous fusion imaging systems, such as those used in UAV / helicopter electro-optical pods and electro-optical search and warning equipment. Continuous zoom heterogeneous fusion imaging systems can determine the field of view control position during the continuous zoom process with pre-calibration. However, due to mechanical and transmission errors in the focusing process, slight mismatches in the dual-channel field of view can occur, reducing the clarity of the fused image and affecting target observation, recognition, and tracking. Therefore, to address the problem of detecting and adjusting slight field of view mismatches during continuous zoom heterogeneous fusion imaging, it is necessary to drive automatic zoom adjustment of a channel based on the alignment between the heterogeneous images after continuous zoom to reduce the field of view mismatch.

[0003] Currently, heterogeneous image registration methods include region-based, feature-based, and deep learning-based methods. Feature-based methods rely on the selection of feature descriptors, such as scale-invariant feature transforms, accelerated robust features, and corner points. However, due to the significant differences between heterogeneous images, it is difficult to generate robust feature descriptors. Deep learning-based methods are generally categorized into feature extraction training, similarity metric learning, direct registration, and modality conversion. However, due to the long execution time and the large number of image pairs required for training, these methods are currently not suitable for application in practical systems. Region-based methods primarily rely on the grayscale characteristics of the image, are sensitive to field of view changes, and offer high registration accuracy. They are widely used for heterogeneous image registration and include mutual information methods, normalized mutual information methods, gradient mutual information methods, improved gradient mutual information methods, and mutual information and edge gradient methods. These mutual information-related methods have achieved some success in the field of medical imaging, but they fail to address the poor grayscale correlation of heterogeneous images in natural environments, especially under small field of view changes, and have certain limitations.

[0004] Currently, automatic zoom search methods include: Fibonacci method, global search method, function approximation method and hill climbing method. Hill climbing search method is one of the most commonly used methods. Its principle is simple and the search speed is fast, but it is easy to fall into local extreme values, resulting in search failure, and certain improvements are needed. Summary of the Invention

[0005] The purpose of the present invention is to provide a heterogeneous image fusion zoom mismatch adjustment method based on edge gradient mutual information. After the heterogeneous fusion imaging system continuously zooms, the edge gradient mutual information is used as an evaluation index of the heterogeneous field of view matching to guide the automatic zoom of one of the lenses, and an improved hill climbing search method is used to search for the maximum value of the evaluation index, thereby solving the problem of slight field of view mismatch in the continuously zooming heterogeneous fusion imaging system.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] The present invention discloses a heterogeneous image fusion zoom mismatch adjustment method based on edge gradient mutual information. The method extracts edges from preprocessed heterogeneous input images, performs normalized mutual information statistics on heterogeneous edge grayscale images, and then performs gradient weighted statistics on edge images, which are used as weights for edge normalized mutual information. Edge gradient mutual information is used as an evaluation index for heterogeneous field of view matching. A hill climbing search method improved by an optimized search direction change strategy is adopted, and three adjacent images are used as criteria to search for the maximum value of the evaluation index, i.e., the best matching position. The method includes the following steps:

[0008] Step 1: Input and preprocessing of heterogeneous image pairs;

[0009] When there are differences between heterogeneous images, preprocessing operations are performed, including: image grayscale, image interpolation and scaling, and image compression;

[0010] As a preferred method, the image grayscale method includes weighted average method, maximum method, and component method; the image interpolation operation includes nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation; the image compression adopts the method based on 3σ in The principle of automatic gain control AGC method is shown in formula (1):

[0011]

[0012] Among them, I in is the input image, I0 is the compressed output image, is the grayscale average of the input image; σ in is the standard deviation of the grayscale of the input image pixels;

[0013] Step 2: Determine the gradient vector and edge extraction of the image;

[0014] Step 2.1: Determine the gradient of each pixel of the heterogeneous image;

[0015] Preferably, the Gaussian gradient operator is used to determine the gradient vector of the image, as shown in formula (2):

[0016]

[0017] in, is the gradient of each point of the heterogeneous image, I(x,y) is the heterogeneous grayscale image, is the Gaussian gradient operator, as shown in formula (3), and Respectively represent the gradients of the heterogeneous image in the x-direction and y-direction, x and y are the horizontal and vertical coordinates of the pixel points of the heterogeneous image, "*" represents convolution, and the superscript T represents the matrix transpose symbol;

[0018]

[0019] Where h(x,y) is a two-dimensional Gaussian function, as shown in formula (4), and are the first-order derivatives of h(x,y) in the x and y directions, respectively, as shown in Equation (5) and Equation (6):

[0020]

[0021] Where σ is the standard deviation of the two-dimensional Gaussian function;

[0022]

[0023]

[0024] Step 2.2: Use constrained gradient to extract edges of heterogeneous images and highlight common edge features. The constrained gradient is shown in formula (7);

[0025]

[0026] Among them, I(x,y) is a heterogeneous grayscale image, E(x,y) is a heterogeneous edge image, is the size of the heterogeneous image gradient, μ is the average value of the heterogeneous image gradient; σ is the standard deviation of the image gradient; a is the coefficient of the standard deviation;

[0027] Step 3: marginal normalized mutual information statistics;

[0028] The normalized mutual information of the edge images of heterogeneous images is statistically analyzed, and the grayscale correlation between the heterogeneous images is statistically expressed as follows:

[0029]

[0030] Among them, EN MI (E A ,E B ) is the edge normalized mutual information of heterogeneous images, H(E A )、H(E B ) are the information entropy of the edge images of heterogeneous image A and heterogeneous image B, H(EA , E B ) is the joint information entropy of the edge images of heterogeneous images A and B, E A 、E B are the edge images of heterogeneous images A and B respectively;

[0031] Step 4: Gradient weighted statistics;

[0032] Step 4.1 Determine the gradient magnitude correlation of each pixel point of the edge image of the heterogeneous image:

[0033] The proportion of the gradient of each pixel of the edge image of the heterogeneous image is used as a coefficient and multiplied by the gradient of the other party, as shown in formula (9):

[0034]

[0035] Among them, M(E A , E B ) is the gradient magnitude correlation of the edge images of heterogeneous images A and B, and are the gradient values ​​of each pixel point of the edge image of heterogeneous image A and the edge image of heterogeneous image B respectively;

[0036] When the difference between the two values ​​is large, the gradient with a smaller value will get a larger proportion; when the difference between the two values ​​is small, both can be considered comprehensively;

[0037] Step 4.2 determines the gradient direction correlation of each pixel point of the edge image of the heterogeneous image, as shown in formula (10):

[0038]

[0039] Among them, W(α) is the gradient direction correlation of the edge images of heterogeneous images A and B, α is the angle of the gradient of each pixel point of the heterogeneous edge image, and are the gradient vectors of each pixel point of the edge image of heterogeneous image A and the edge image of heterogeneous image B respectively;

[0040] Step 4.3 considers the size and direction of the gradient comprehensively, directly determines the gradient weight value of the edge image of the heterogeneous image, and obtains the gradient weight G(E A ,E B ), as shown in formula (11):

[0041]

[0042] Step 5: Edge gradient mutual information value statistics;

[0043] For the edge images of heterologous images, the gradient information of the edge images is used as the weight of the edge normalized mutual information to obtain the edge gradient mutual information value, as shown in Equation (12), to achieve the fusion of gradient and gray-scale information;

[0044] EG NMI (E A ,E B ) = G(E A ,E B ) × EN MI (E A ,E B ) (12)

[0045] Among them, EG MI (E A ,E B ) is the edge gradient mutual information of the heterologous edge images E A and E B , EN MI (E A ,E B ) is the normalized mutual information of the heterologous edge images E A and E B ; G(E A ,E B ) is the edge gradient weighting;

[0046] Step Six Search for the best matching position;

[0047] Based on the hill-climbing search method that optimizes the search direction change strategy, using three adjacent heterologous images as the criterion, search for the maximum value of the evaluation index edge gradient mutual information, that is, the best matching position of the heterologous fields of view, specifically including the following sub-steps:

[0048] Step 6.1 Collect heterologous images and count the edge gradient mutual information values:

[0049] Specify the search step size step and the search direction, and continuously collect three pairs of heterologous images, count the edge gradient mutual information values of the three pairs of heterologous images, and the edge gradient mutual information values of the three pairs of heterologous images are F1, F2, and F3 respectively;

[0050] Step 6.2 Compare the evaluation index values and determine the search direction:

[0051] When F1 < F2 < F3, the search direction is correct, keep the search direction unchanged and continue to search; when F1 > F2 < F3, a minimum point appears, it is recommended to continue to search along the search direction and observe the subsequent curve trend; when F1 > F2 > F3, the search direction is incorrect, reverse the search direction and continue to search;

[0052] Step 6.3 Judge the pseudo-peak point and determine the search direction and search step size:

[0053] When F1 <f2>When F3 reaches the peak point, to avoid false peak points, continue to collect a heterologous image along the search direction with the search step size step, and determine the edge gradient mutual information value F4;

[0054] When F3 > F4, reverse the search direction, change the search step size step to step / 2, and continue the search;

[0055] When F3 < F4, determine that the peak point is a false peak point, avoid the false peak point, and continue to maintain the search direction;

[0056] Step 6.4 Zooming ends;

[0057] When the search step size is less than the threshold δ, it is considered that the current peak point is the best zoom position, and the corresponding one is the best matching position of the field of view;

[0058] Preferably, the threshold δ is determined according to the actual accuracy requirements and system speed;

[0059] Step Seven Confirm the best zoom position of the driving lens and complete the field of view error correction.

[0060] Beneficial effects:

[0061] 1. The method for adjusting zoom mismatch of heterologous image fusion based on edge gradient mutual information of the present invention combines heterologous image registration and automatic zooming. After continuous zooming of the heterologous fusion imaging system, the edge gradient mutual information is used as an evaluation index for field of view matching, and an optimized hill climbing search method is used to search for the maximum value of the edge gradient mutual information, solving the problem of slight field of view mismatch existing in the continuous zooming heterologous fusion imaging system, and realizing the application of the continuous zooming heterologous fusion imaging system in fields such as UAV / helicopter optoelectronic pods, optoelectronic search and warning equipment, and security monitoring.

[0062] 2. The method for adjusting zoom mismatch of heterologous image fusion based on edge gradient mutual information of the present invention, based on the evaluation index of heterologous field of view matching of edge gradient mutual information, combines the gradient information and gray information of the edge image, performs normalized mutual information statistics on the edge images of heterologous images, improves the gray correlation of heterologous images, constructs a gradient weighting sensitive to tiny field of view changes, thereby enhancing the unimodality and sensitivity of the evaluation index.

[0063] 3. The method for adjusting zoom mismatch of heterologous image fusion based on edge gradient mutual information of the present invention uses an optimized hill climbing search method, takes the adjacent three heterologous images as criteria, compares the edge gradient mutual information values of the three heterologous images, guides the change of the search direction, when a peak point appears, continues to collect the next heterologous image according to the step size, and compares its edge gradient mutual information value with the previous heterologous image to determine whether the peak point is the global maximum point, which can effectively suppress the influence of local extreme values and improve the zoom registration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 are the original heterogeneous images, where: (a) is the fixed-focus infrared image, (b) is the visible light image during the zoom process, and (c) is the fused image during the zoom process;

[0065] Figure 2 This is a flow chart of the zoom mismatch adjustment method for heterogeneous image fusion based on edge gradient mutual information of the present invention;

[0066] Figure 3 are the gradient and edge images of the infrared image and the visible light image, where: (a) is the original visible light image and the original infrared image, (b) is the gradient map of the visible light image and the gradient map of the infrared image, and (c) is the edge image of the visible light image and the edge image of the infrared image;

[0067] Figure 4 A flowchart of the improved hill climbing search method of the present invention; Figure 5 is the fused image at the best match.

[0068] Figure 6 Comparison graphs of different evaluation index curves, where: (a) is the normalized mutual information curve, (b) is the gradient mutual information curve, (c) is the improved gradient mutual information curve, (d) is the edge normalized mutual information curve, (e) is the mutual information and edge gradient curve, and (f) is the edge gradient mutual information curve of the present invention. DETAILED DESCRIPTION

[0069] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below with reference to the accompanying drawings and examples.

[0070] Example 1:

[0071] This example uses a dual-band fusion system to capture a fixed-focus longwave infrared image sequence and a zoomed visible light image sequence. The dual-band fusion system primarily consists of a fixed-focus longwave infrared lens and a zoomed visible light lens. It simulates continuous zoom and then drives the visible light lens to automatically adjust its focus, minimizing differences in the fields of view between the two sources.

[0072] In the dual-band fusion system of the embodiment, the focal length of the long-wave infrared objective lens is 40 mm, the detector is an uncooled focal plane detector of Yantai Arrow, the number of pixels is 640×512, and the image is 14 bits; the focal length of the visible light objective lens is 12.5-75 mm, the detector is a low-illumination CMOS movement of Kunshan Ruixin Micro, the number of pixels is 1280×1024, and the image is 8 bits.

[0073] like Figure 1 As shown, the dual-band fusion system of the embodiment is used to collect 50 sets of fixed-focus infrared images and zoom visible light image sequences in one scene. Figure 1 (a) is a fixed-focus infrared image. Figure 1 (b) is the visible light image during the zoom process. Figure 1 (c) is the grayscale fusion image based on color transfer, such as Figure 1 As shown in (c), there is a slight difference in field of view between the visible light image and the infrared image. As the focal length of the visible light lens is fine-tuned, the field of view difference gradually decreases. After passing the optimal matching position of the field of view, the field of view difference gradually increases.

[0074] The embodiment applies the heterogeneous image fusion zoom mismatch adjustment method based on edge gradient mutual information of the present invention to simulate the image fusion of fixed-focus infrared images and zoom visible light image sequences collected by the dual-band fusion system. Figure 3 As shown, the following steps are included:

[0075] Step 1: Input and preprocessing of heterogeneous image pairs;

[0076] In the embodiment, there are differences between the visible light image and the infrared image, and preprocessing operations are required, including: grayscale, interpolation scaling and image compression;

[0077] The weighted average method is used to grayscale the color visible light image, and the bicubic interpolation method is used to interpolate and scale the 1280×1024 visible light image to a 640×512 image.

[0078] The 3σ-based in The automatic gain control AGC method based on the principle, as shown in formula (1), compresses the 14-bit infrared image into an 8-bit image;

[0079]

[0080] Among them, I in is the input 14-bit infrared image, I0 is the compressed output 8-bit infrared image, is the grayscale average of the input image, σ in is the standard deviation of the grayscale of the input image pixels;

[0081] Step 2: Determine the gradient vector and edge extraction of the image;

[0082] Step 2.1: Determine the gradient of each pixel of the heterogeneous image;

[0083] In the embodiment, the Gaussian gradient operator is used to determine the gradient vector of the image, as shown in formula (2):

[0084]

[0085] in, is the gradient of each point of the heterogeneous image, I(x,y) is the heterogeneous grayscale image, is the Gaussian gradient operator, as shown in formula (3), and Respectively represent the gradients of the heterogeneous image in the x-direction and y-direction, x and y are the horizontal and vertical coordinates of the pixel points of the heterogeneous image, "*" represents convolution, and the superscript T represents the matrix transpose symbol;

[0086]

[0087] Where h(x,y) is a two-dimensional Gaussian function, as shown in formula (4), and are the first-order derivatives of h(x,y) in the x and y directions, respectively, as shown in Equation (5) and Equation (6):

[0088]

[0089] Where σ is the standard deviation of the two-dimensional Gaussian function;

[0090]

[0091]

[0092] In the embodiment, the original visible light image and the original infrared image are as follows: Figure 3 As shown in (a), the gradient vectors of the visible light image and the infrared image are determined using a 5×5 Gaussian gradient operator template with σ=1, as shown in Figure 3 (b)

[0093] Step 2. Use constrained gradient to extract edges of heterogeneous images and highlight common edge features. The constrained gradient is shown in formula (7);

[0094]

[0095] Among them, I(x,y) is a heterogeneous grayscale image, E(x,y) is a heterogeneous edge image, is the size of the heterogeneous image gradient, μ is the average value of the heterogeneous image gradient; σ is the standard deviation of the image gradient; a is the coefficient of the standard deviation;

[0096] In the embodiment, the value of a for edge extraction of visible light image is 0.25, and the value of a for edge extraction of infrared image is 0. Figure 3 (c) shows the edge image of the visible light image and the edge image of the infrared image;

[0097] Step 3: marginal normalized mutual information statistics;

[0098] The normalized mutual information of the edge images of heterogeneous images is statistically analyzed, and the grayscale correlation between the heterogeneous images is statistically expressed as follows:

[0099]

[0100] Among them, EN MI (E A ,E B ) is the edge normalized mutual information of heterogeneous images, H(E A )、H(E B ) are the information entropy of the edge images of heterogeneous image A and heterogeneous image B, H(E A , E B ) is the joint information entropy of the edge images of heterogeneous images A and B, E A 、E B are the edge images of heterogeneous images A and B respectively;

[0101] Step 4: Gradient weighted statistics;

[0102] Step 4.1 Determine the gradient magnitude correlation of each pixel point of the edge image of the heterogeneous image:

[0103] The proportion of the gradient of each pixel of the edge image of the heterogeneous image is used as a coefficient and multiplied by the gradient of the other party, as shown in formula (9):

[0104]

[0105] Among them, M(E A , E B ) is the gradient magnitude correlation of the edge images of heterogeneous images A and B, and are the gradient values ​​of each pixel point of the edge image of heterogeneous image A and the edge image of heterogeneous image B respectively;

[0106] When the difference between the two values ​​is large, the gradient with a smaller value will get a larger proportion; when the difference between the two values ​​is small, both can be considered comprehensively;

[0107] Step 4.2, determine the gradient direction correlation of each pixel point of the edge image of the heterogeneous image, as shown in formula (10):

[0108]

[0109] Among them, W(α) is the gradient direction correlation of the edge images of heterogeneous images A and B, α is the angle of the gradient of each pixel point of the heterogeneous edge image, and are the gradient vectors of each pixel point of the edge image of heterogeneous image A and the edge image of heterogeneous image B respectively;

[0110] Step 4.3 considers the size and direction of the gradient comprehensively, directly determines the gradient weight value of the edge image of the heterogeneous image, and obtains the gradient weight G(E A ,E B ), as shown in formula (11):

[0111]

[0112] Step 5: Edge gradient mutual information value statistics;

[0113] For the edge images of visible light images and infrared images, the edge image gradient information is used as the weight of the edge normalized mutual information to obtain the edge gradient mutual information value, as shown in formula (12), to achieve the fusion of gradient and grayscale information;

[0114] EG NMI (E A ,E B )=G(E A ,E B )×EN MI (E A ,E B ) (12)

[0115] Among them, EG MI (E A ,E B ) is the heterogeneous edge image E A and E B The edge gradient mutual information, EN MI (E A ,E B ) is the heterogeneous edge image E A and E B Normalized mutual information of G(E A ,E B ) is the edge gradient weight;

[0116] Step 6: Search for the best matching position;

[0117] The hill climbing search method based on the optimization search direction change strategy uses three adjacent heterogeneous images as the criterion to search for the maximum value of the evaluation index edge gradient mutual information, that is, the best matching position of the heterogeneous field of view, such as Figure 4 As shown, it specifically includes the following sub-steps:

[0118] Step 6.1: Collect heterogeneous images and calculate the edge gradient mutual information value:

[0119] Specify the search step size and search direction, and continuously collect three pairs of heterogeneous images. Count the edge gradient mutual information values ​​of the three pairs of heterogeneous images. The edge gradient mutual information values ​​of the three pairs of heterogeneous images are F1, F2, and F3 respectively.

[0120] Step 6.2, compare the evaluation index values and determine the search direction:

[0121] When F1 < F2 < F3, the search direction is correct, keep the search direction unchanged and continue the search; when F1 > F2 < F3, a minimum point appears, it is recommended to continue the search along the search direction and observe the subsequent curve trend; when F1 > F2 > F3, the search direction is incorrect, reverse the search direction and continue the search;

[0122] Step 6.3, determine the pseudo-peak point and decide the search direction and search step size:

[0123] When F1 <f2>When F3, a peak point appears. To avoid false peak points, continue to collect a heterologous image along the search direction with the search step size step, and determine the edge gradient mutual information value F4;

[0124] When F3 > F4, reverse the search direction, change the search step size step to step / 2, and continue the search;

[0125] When F3 < F4, determine that the peak point is a false peak point, avoid the false peak point, and continue to maintain the search direction;

[0126] Step 6.4, the zoom ends;

[0127] When the search step size is less than the threshold δ, it is considered that the current peak point is the best zoom position, and the corresponding one is the best matching position of the field of view;

[0128] Preferably, the threshold δ is determined according to the actual accuracy requirements and system speed;

[0129] Step seven, confirm the best zoom position of the driving lens, and complete the correction of the field of view error;

[0130] In the embodiment, drive the visible light lens to the best zoom position, complete the correction of the error between the infrared field of view and the visible light field of view, realize the image fusion of the fixed-focus infrared image sequence and the zoom visible light image sequence. The initial search step size step is set to 1 mm, the step size threshold δ is set to 0.1 mm, and the fused image at the best matching position is as Figure 5 shown.

[0131] In the embodiment, the curves of different evaluation indexes of the visible light image and the infrared image under different methods are respectively as Figure 6 (a)-(f) shown, Figure 6 (a) is the normalized mutual information curve graph, Figure 6 (b) is the gradient mutual information curve graph, Figure 6 (c) is the improved gradient mutual information curve graph, Figure 6 (d) is the edge normalized mutual information curve graph, Figure 6 (e) is the mutual information and edge gradient curve graph, Figure 6 (f) is the edge gradient mutual information curve graph of the present invention. Compared with other several evaluation methods, the edge gradient mutual information evaluation index of the present invention has better curve quality, better unimodality, micro field of view sensitivity and robustness;

[0132] On the other hand, the improved hill climbing search method can suppress local extrema, accurately find the peak value of the edge gradient mutual information evaluation index, and realize that after continuous zooming of the dual-band fusion system, the visible light objective lens is driven to automatically zoom to reduce the field of view difference, thereby solving the problem of tiny mismatch in heterologous field of view zooming.

[0133] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A zoom mismatch adjustment method for heterogeneous image fusion based on edge gradient mutual information, characterized by: Edge extraction is performed on the preprocessed heterologous input image, normalized mutual information statistics are carried out on the heterologous edge grayscale image, and then gradient weighting statistics are performed on the edge image, which is used as the weighting of the edge normalized mutual information. The edge gradient mutual information is used as the evaluation index for the matching situation of the heterologous fields of view. An improved hill climbing search method with an optimized search direction change strategy is adopted, and the adjacent three images are used as criteria to search for the maximum value of the evaluation index, that is, the most matching position, including the following steps: Step 1: Input and preprocessing of heterologous image pairs; When there are differences between heterologous images, preprocessing operations are carried out, including: image grayscale conversion, image interpolation operation, and image compression; Step 2: Determine the gradient vector and edge extraction of the image; Step 2.1: Determine the gradient of each pixel point of the heterologous image; Step 2.2: Perform edge extraction on the heterologous images respectively through constrained gradients to highlight the common edge features. The constrained gradient is shown in Equation (1); Among them, I(x,y) is a heterogeneous grayscale image, E(x,y) is a heterogeneous edge image, is the size of the heterogeneous image gradient, x and y are the horizontal and vertical coordinates of the heterogeneous image pixel points, μ is the average value of the heterogeneous image gradient; σ is the standard deviation of the image gradient; a is the coefficient of the standard deviation; Step 3: Edge normalized mutual information statistics; Perform normalized mutual information statistics on the edge images of heterologous images, and statistically analyze the gray correlation between heterologous images as shown in Equation (2): Among them, EN MI (E A ,E B ) is the edge normalized mutual information of heterogeneous images, H(E A )、H(E B ) are the information entropy of the edge images of heterogeneous image A and heterogeneous image B, H(E A , E B ) is the joint information entropy of the edge images of heterogeneous images A and B, E A 、E B are the edge images of heterogeneous images A and B respectively; Step 4: Gradient weighting statistics; Step 4.1: Determine the gradient magnitude correlation of each pixel point of the edge image of the heterologous image: Taking the proportion of the gradient of each pixel point of the edge image of the heterologous image as a coefficient, multiply it with the gradient of the other party respectively, as shown in Equation (3): Among them, M(E A , E B ) is the gradient magnitude correlation of the edge images of heterogeneous images A and B, and are the gradient values ​​of each pixel point of the edge image of heterogeneous image A and the edge image of heterogeneous image B respectively; When the two values differ greatly, the gradient with a smaller value will obtain a larger proportion; when the difference between the two is small, both can be comprehensively considered; Step 4.2: Determine the gradient direction correlation of each pixel point of the edge image of the heterologous image, as shown in Equation (4): Among them, W(α) is the gradient direction correlation of the edge images of heterogeneous images A and B, α is the angle of the gradient of each pixel point of the heterogeneous edge image, and are the gradient vectors of each pixel point of the edge image of heterogeneous image A and the edge image of heterogeneous image B respectively; Step 4.3 considers the size and direction of the gradient comprehensively, directly determines the gradient weight value of the edge image of the heterogeneous image, and obtains the gradient weight G(E A ,E B ), as shown in formula (5): Step 5: Edge gradient mutual information value statistics; For the edge images of heterologous images, use the gradient information of the edge image as the weighting of the edge normalized mutual information to obtain the edge gradient mutual information value, as shown in Equation (6), to achieve the fusion of gradient and gray information; EG NMI (E A ,E B )=G(E A ,E B )×EN MI (E A ,E B ) (6) Among them, EG NMI (E A ,E B ) is the heterogeneous edge image E A and E B The edge gradient mutual information, EN MI (E A ,E B ) is the heterogeneous edge image E A and E B Normalized mutual information of G(E A ,E B ) is the edge gradient weight; Step 6: Search for the best matching position; Based on the hill climbing search method with an optimized search direction change strategy, using the adjacent three heterologous images as criteria, search for the maximum value of the evaluation index edge gradient mutual information, that is, the best matching position of the heterologous fields of view, specifically including the following sub-steps: Step 6.1: Collect heterologous images and statistically analyze the edge gradient mutual information value: Specify the search step size step and the search direction, and continuously collect three pairs of heterologous images, and statistically analyze the edge gradient mutual information values of the three pairs of heterologous images. The edge gradient mutual information values of the three pairs of heterologous images are F1, F2, and F3 respectively; Step 6.2: Compare the evaluation index values and determine the search direction: When F1 < F2 < F3, the search direction is correct, and continue to search in the same search direction; when F1 > F2 < F3, a minimum value point appears, it is recommended to continue to search along the search direction and observe the subsequent curve trend; when F1 > F2 > F3, the search direction is incorrect, reverse the search direction and continue to search; Step 6.3: Judge the pseudo peak point and determine the search direction and search step size: When F1 <f2>When F3 appears, a peak point appears. To avoid the pseudo peak point, continue to collect a heterologous image along the search direction with the search step size step, and determine the edge gradient mutual information value F4; When F3 > F4, reverse the search direction, change the search step size step to step / 2, and continue the search; When F3 < F4, determine that the peak point is a false peak point, avoid the false peak point, and continue to maintain the search direction; Step 6.4 Zoom ends; When the search step size is less than the threshold δ, it is considered that the current peak point is the optimal zoom position, and the corresponding one is the best matching position of the field of view; Step Seven Confirm the optimal zoom position of the driving lens and complete the field of view error correction.

2. The zoom mismatch adjustment method for heterogeneous image fusion based on edge gradient mutual information according to claim 1, characterized in that: Image grayscale methods include weighted average method, maximum method, and component method; image interpolation operations include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation; image compression uses 3σ-based in The automatic gain control AGC method based on the principle is shown in formula (7): Among them, I in is the input image, I0 is the compressed output image, is the grayscale average of the input image; σ in is the standard deviation of the grayscale of the input image pixels.

3. The zoom mismatch adjustment method for heterogeneous image fusion based on edge gradient mutual information according to claim 1, characterized in that: Use the Gaussian gradient operator to determine the gradient vector of the image, as shown in Equation (8): in, is the gradient of each point of the heterogeneous image, I(x,y) is the heterogeneous grayscale image, is the Gaussian gradient operator, as shown in formula (9), and They represent the gradients of the heterogeneous image in the x-direction and y-direction respectively, "*" represents convolution, and the superscript T represents the matrix transpose symbol; Where h(x,y) is a two-dimensional Gaussian function, as shown in formula (10), and are the first-order derivatives of h(x,y) in the x and y directions, respectively, as shown in Equation (11) and Equation (12): Where, σ is the standard variance of the two-dimensional Gaussian function; 4. The zoom mismatch adjustment method for heterogeneous image fusion based on edge gradient mutual information according to claim 1, characterized in that: The threshold δ is determined according to the actual accuracy requirements and system speed.

Images