A heterogeneous image matching method, device, equipment and storage medium

Abstract

The application discloses a heterogeneous image matching method and device, equipment and storage medium, and relates to the technical field of image processing. The method comprises the following steps: obtaining initial matching parameters of a reference image and a to-be-matched image, and generating target matching parameters; determining an overlapping image region between the reference image and the to-be-matched image based on the initial matching parameters; determining a reference image overlapping region in the reference image and a to-be-matched image overlapping region in the to-be-matched image based on the overlapping region; constructing an image optimization function by using a gray similarity function constructed based on the reference image overlapping region and the to-be-matched image overlapping region; performing iterative processing on the image optimization function based on the target matching parameters to determine optimal matching parameters; and performing affine transformation on the to-be-matched image by using the optimal matching parameters, and outputting a target image after matching. The method solves the problem of low matching precision of heterogeneous images caused by large gray difference and scale difference.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for heterogeneous image matching. Background Technology

[0002] With the rapid development of imaging sensor technology and the increasing complexity of application environments, single-source imaging sensors are now struggling to meet practical application needs. Heterogeneous imaging sensors, on the other hand, can effectively process information from multiple imaging sensors, increasing the usefulness and richness of the information. Accurate matching of heterogeneous images is a prerequisite for collaborative processing of heterogeneous images.

[0003] Currently, heterogeneous image matching methods are generally divided into feature-based matching methods and gray-level-based matching methods. Feature-based registration methods have clear advantages: they are robust to scale and rotation transformations, have low computational cost, and high accuracy. However, for heterogeneous images with significant gray-level differences, it is difficult to find enough high-precision feature matching pairs, and the matching accuracy of direct feature-based matching methods is insufficient. Gray-level-based methods utilize the statistical characteristics of global gray-level distribution to construct similarity metrics, and then use appropriate optimization algorithms to find the optimal registration parameters. Gray-level-based methods are robust to differences in gray-level and texture, but cannot handle scale differences. Addressing the significant differences in gray-level and scale in heterogeneous images, many researchers have proposed remote sensing image registration methods that combine gray-level and feature-based approaches. Feature-based matching methods are used for coarse matching, and then gray-level-based methods are used to optimize the registration parameters.

[0004] In summary, how to improve the accuracy and robustness of heterogeneous image matching and reduce the low accuracy of heterogeneous image matching caused by large differences in grayscale and scale between heterogeneous images is a technical problem that needs to be solved in this field. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for heterogeneous image matching, which can improve the accuracy and robustness of heterogeneous image matching and reduce the low matching accuracy caused by large differences in grayscale and scale between heterogeneous images. The specific solution is as follows:

[0006] In a first aspect, this application discloses a heterogeneous image matching method, including:

[0007] Obtain initial matching parameters for the reference image and the image to be matched, and generate target matching parameters based on the initial matching parameters;

[0008] Based on the initial matching parameters, the overlapping image region between the reference image and the image to be matched is determined;

[0009] Based on the overlapping region, a reference image overlapping region is determined in the reference image, and a matching image overlapping region is determined in the image to be matched.

[0010] An image optimization function is constructed using a gray-level similarity function based on the overlapping regions of the reference image and the overlapping regions of the image to be matched.

[0011] The image optimization function is iteratively processed based on the target matching parameters to determine the optimal matching parameters;

[0012] The optimal matching parameters are used to perform an affine transformation on the image to be matched, and the matched target image is output.

[0013] Optionally, the initial matching parameters based on the obtained reference image and the image to be matched include:

[0014] The scale ratio between the reference image and the image to be matched is calculated based on the camera focal length and camera parameters of the reference image and the image to be matched.

[0015] The initial matching parameters are determined based on the scale ratio, the length and width of the reference image, and the length and width of the image to be matched.

[0016] Optionally, determining the overlapping image region between the reference image and the image to be matched based on the initial matching parameters includes:

[0017] The target coordinates in the corresponding reference image are determined using a preset coordinate transformation formula and the vertex coordinates of the image to be matched.

[0018] The horizontal and vertical coordinates of the target coordinates are rounded and sorted to determine the position information of the overlapping image region;

[0019] Based on the location information, the overlapping image region between the reference image and the image to be matched is determined.

[0020] Optionally, the process of constructing the image optimization function using the gray-level similarity function based on the overlapping regions of the reference image and the overlapping regions of the image to be matched further includes:

[0021] A gray-scale statistical similarity function is constructed based on the entropy of the overlapping region of the reference image, the entropy of the overlapping region of the image to be matched after correction by the target matching parameters, and the joint entropy of the overlapping region of the reference image and the overlapping region of the image to be matched.

[0022] Optionally, the process of constructing the image optimization function using the gray-level similarity function based on the overlapping regions of the reference image and the overlapping regions of the image to be matched further includes:

[0023] A gray-scale structural similarity function is constructed based on the mean gray-scale value of the overlapping region of the reference image, the mean gray-scale value of the overlapping region of the fused image to be matched, the gray-scale variance of the overlapping region of the reference image, and the gray-scale variance of the overlapping region of the fused image to be matched.

[0024] Optionally, the heterogeneous image matching method further includes:

[0025] The fused overlapping region of the image to be matched is determined based on the overlapping region of the reference image and the corrected overlapping region of the image to be matched.

[0026] Optionally, the iterative processing of the image optimization function based on the target matching parameters to determine the optimal matching parameters includes:

[0027] Calculate the current similarity function value of the image optimization function, and compare the current similarity function value with the target similarity function value;

[0028] If the current similarity function value is less than the target similarity function value, then the current similarity function value is updated to the target similarity function value;

[0029] The process involves calculating the current similarity function value of the image optimization function until the current iteration count meets the preset iteration count, in order to obtain the optimal matching parameters.

[0030] Secondly, this application discloses a heterogeneous image matching device, comprising:

[0031] An initial parameter acquisition module is used to acquire initial matching parameters of a reference image and an image to be matched, and to generate target matching parameters based on the initial matching parameters;

[0032] The overlapping region determination module is used to determine the overlapping image region between the reference image and the image to be matched based on the initial matching parameters;

[0033] An image determination module is used to determine a reference image overlapping region in the reference image based on the overlapping region, and to determine a matching image in the image to be matched.

[0034] An optimization function construction module is used to construct an image optimization function using a gray-level similarity function based on the overlapping regions of the reference image and the overlapping regions of the image to be matched.

[0035] The optimal parameter determination module is used to iteratively process the image optimization function based on the target matching parameters to determine the optimal matching parameters;

[0036] The matching image acquisition module is used to perform an affine transformation on the image to be matched using the optimal matching parameters, and output the matched target image.

[0037] Thirdly, this application discloses an electronic device, including:

[0038] Memory, used to store computer programs;

[0039] A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed heterogeneous image matching method.

[0040] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed heterogeneous image matching method.

[0041] Therefore, this application discloses a heterogeneous image matching method, comprising: obtaining initial matching parameters of a reference image and an image to be matched, and generating target matching parameters based on the initial matching parameters; determining the overlapping image region between the reference image and the image to be matched based on the initial matching parameters; determining a reference image overlapping region in the reference image and a matching image overlapping region in the image to be matched based on the overlapping region; constructing an image optimization function using a gray-level similarity function constructed based on the reference image overlapping region and the matching image overlapping region; iteratively processing the image optimization function based on the target matching parameters to determine the optimal matching parameters; and performing an affine transformation on the image to be matched using the optimal matching parameters to output the matched target image. It is evident that by comprehensively considering the influence of the similarity metric constructed from gray-level similarity on heterogeneous image matching, solving the image optimization function constructed therefrom, and determining the optimal matching parameters through population optimization, the accuracy and robustness of heterogeneous image matching are improved, solving the problem of low accuracy in heterogeneous image matching caused by large gray-level and scale differences. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0043] Figure 1 This is a flowchart of a heterogeneous image matching method disclosed in this application;

[0044] Figure 2 This is a reference image disclosed in this application;

[0045] Figure 3 This application discloses a corrected image to be matched.

[0046] Figure 4 This is a flowchart of a specific heterogeneous image matching method disclosed in this application;

[0047] Figure 5 This application discloses an image to be matched after an initial spatial transformation.

[0048] Figure 6 This is a fused image obtained by optimizing the initial matching parameters of an aerial photograph of a photovoltaic module as disclosed in this application;

[0049] Figure 7 This application discloses a fused image obtained by matching aerial images of photovoltaic modules with optimal matching parameters.

[0050] Figure 8 This is a flowchart of another specific heterogeneous image matching method disclosed in this application;

[0051] Figure 9 This is a schematic diagram of the structure of a heterogeneous image matching device disclosed in this application;

[0052] Figure 10 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0054] With the rapid development of imaging sensor technology and the increasing complexity of application environments, single-source imaging sensors are now struggling to meet practical application needs. Heterogeneous imaging sensors, on the other hand, can effectively process information from multiple imaging sensors, increasing the usefulness and richness of the information. Accurate matching of heterogeneous images is a prerequisite for collaborative processing of heterogeneous images.

[0055] Currently, heterogeneous image matching methods are generally divided into feature-based matching methods and gray-level-based matching methods. Feature-based registration methods have clear advantages: they are robust to scale and rotation transformations, have low computational cost, and high accuracy. However, for heterogeneous images with significant gray-level differences, it is difficult to find enough high-precision feature matching pairs, and the matching accuracy of direct feature-based matching methods is insufficient. Gray-level-based methods utilize the statistical characteristics of global gray-level distribution to construct similarity metrics, and then use appropriate optimization algorithms to find the optimal registration parameters. Gray-level-based methods are robust to differences in gray-level and texture, but cannot handle scale differences. Addressing the significant differences in gray-level and scale in heterogeneous images, many researchers have proposed remote sensing image registration methods that combine gray-level and feature-based approaches. Feature-based matching methods are used for coarse matching, and then gray-level-based methods are used to optimize the registration parameters.

[0056] To address this, this application discloses a heterogeneous image matching scheme that can improve the accuracy and robustness of heterogeneous image matching, and reduce the problems of low accuracy and low robustness caused by large differences in grayscale and scale between heterogeneous images.

[0057] Reference Figure 1 As shown, this embodiment of the invention discloses a heterogeneous image matching method, including:

[0058] Step S11: Obtain the initial matching parameters of the reference image and the image to be matched, and generate the target matching parameters based on the initial matching parameters.

[0059] In this embodiment, the initial matching parameters are obtained by feature matching or based on image imaging parameters and imaging characteristics. It is important to note that when the grayscale difference between the reference image and the image to be matched is small, a feature matching method is used to determine the initial matching parameters. This feature matching method can include, but is not limited to, SIFT or SURF. When the grayscale difference between the reference image and the image to be matched is too large, it is difficult to find a sufficient number of high-precision feature matching pairs. In this case, the initial matching parameters are generally determined using image imaging characteristics and imaging parameters. This solves the problem of coarse matching of heterogeneous images when feature matching fails.

[0060] In this embodiment, target matching parameters are generated based on initial matching parameters. The initial matching parameters are regarded as initial particles, and n sets of matching parameters are constructed within a certain range of the initial particles using a random perturbation method. A population is formed using a set of matching parameters; the number of iterations is initialized to t = 0. It can be seen that random perturbation is used to expand the data, i.e., the data is allowed to fluctuate within a certain range, thereby increasing the data volume and improving the robustness of the algorithm. The specific range can be determined by the user based on their actual situation.

[0061] Step S12: Determine the overlapping image region between the reference image and the image to be matched based on the initial matching parameters.

[0062] In this embodiment, the overlapping region of the heterogeneous matching images used for similarity measurement during the parameter optimization process is determined based on the initial matching parameters; wherein, the heterogeneous matching images are as follows: Figure 2 The reference image and the image to be matched shown can be understood as follows: the overlapping area is determined based on the initial matching parameters, which is to say, the result image after coarse matching of the heterogeneous matching images is obtained. In other words, the coarse matching part of the reference image and the image to be matched is obtained as the overlapping area used for similarity measurement. It can be understood that the similarity measurement is to comprehensively evaluate the degree of similarity between two things. Specifically, when performing image matching on two images, the similarity is often measured by the overlapping area between them.

[0063] Step S13: Based on the overlapping region, determine the reference image overlapping region in the reference image and the image to be matched overlapping region in the image to be matched.

[0064] In this embodiment, the target coordinates in the corresponding reference image are determined using a preset coordinate transformation formula and the vertex coordinates of the image to be matched. The horizontal and vertical coordinates of the target coordinates are then rounded and sorted to determine the position information of the overlapping image region. Based on this position information, the overlapping image region between the reference image and the image to be matched is determined. It can be understood that the target coordinates are calculated by taking the coordinates of the four vertices of the image to be matched in the reference image, and the position information of the overlapping image region is determined based on these target coordinates. Specifically, the target coordinates of the four vertices of the image to be matched in the reference image are determined using a coordinate transformation formula, as follows:

[0065]

[0066] The coordinates of the four vertices of the image to be matched are: x ro1 =1,y ro1 =1; x ro2 =H r ,y ro2 =1; x ro3 =1,y ro3 =W r ;x ro4 =H r ,y ro4 =W r ; respectively for the four x-coordinates soj and 4 ordinates y sojAfter rounding and sorting, determine the vertex coordinates of the smallest inscribed rectangle. The first reference image and the corrected image lines and Between lines, the first and The region between columns is the overlapping region of the matched images, where W r H r W represents the length and width of the reference image, respectively. I H I ...

[0067] Step S14: Construct an image optimization function using a grayscale similarity function based on the overlapping regions of the reference image and the overlapping regions of the image to be matched.

[0068] In this embodiment, an image optimization function is constructed using a gray-level similarity function based on the overlapping regions of the reference image and the overlapping regions of the image to be matched. Specifically, the image optimization function is constructed based on gray-level statistical similarity and gray-level structural similarity. Specifically, a gray-level statistical similarity function, i.e., a gray-level statistical similarity measure, is determined based on the overlapping regions of the reference image and the corrected overlapping regions of the image to be matched, serving as one indicator for image registration. A gray-level structural similarity function, i.e., a gray-level structural similarity measure, is determined based on the overlapping regions of the reference image and the fused overlapping regions of the image to be matched, serving as another indicator for image registration. Then, an image optimization function for heterogeneous image registration is constructed using the two image registration indicators mentioned above. The corrected image to be matched is the image generated during the image correction process based on the image to be matched. The fused overlapping region of the image to be matched is determined based on the overlapping regions of the reference image and the corrected overlapping regions of the image to be matched; that is, the fused overlapping region of the image to be matched is the image generated based on the affine transformation between the corrected image to be matched and the reference image. The formula for the optimization function is as follows:

[0069]

[0070] The reference image is denoted as I. r The image to be matched is I s A population is formed by a set of matching parameters, and the image to be matched is I. s The corrected image to be matched is obtained by the affine transformation of the i-th population after the t-th iteration. The corresponding fusion result image is as follows Let represent the similarity function value obtained by the i-th population (i = 1, ..., n) after the t-th iteration for image matching. The gray-level statistical similarity function is used to compare the overlapping regions of the reference image and the corrected image to be matched. This is a grayscale structure similarity function for the overlapping regions of the reference image and the matched fused image.

[0071] Step S15: Iterate the image optimization function based on the target matching parameters to determine the optimal matching parameters.

[0072] In this embodiment, an optimization algorithm and a target matching parameter optimization function are used for iterative solution. After each iteration, the image to be matched is adjusted and calibrated accordingly until the iteration result is the optimal result. The corresponding target matching parameters are then output as the optimal matching parameters. The optimization algorithm may include, but is not limited to, particle swarm optimization (PSO), quantum particle swarm optimization (QPSO), or ant colony optimization.

[0073] Step S16: Apply the optimal matching parameters to the image to be matched using an affine transformation, and output the matched target image.

[0074] In this embodiment, the optimal matching parameters are substituted, and an affine transformation is performed on the image to be matched, outputting as follows: Figure 3 The image shown is the corrected image to be matched and the image of the matching and fusion result.

[0075] Therefore, this application discloses a heterogeneous image matching method, comprising: obtaining initial matching parameters of a reference image and an image to be matched, and generating target matching parameters based on the initial matching parameters; determining the overlapping image region between the reference image and the image to be matched based on the initial matching parameters; determining a reference image overlapping region in the reference image and a matching image overlapping region in the image to be matched based on the overlapping region; constructing an image optimization function using a gray-level similarity function constructed based on the reference image overlapping region and the matching image overlapping region; iteratively processing the image optimization function based on the target matching parameters to determine the optimal matching parameters; and performing an affine transformation on the image to be matched using the optimal matching parameters to output the matched target image. It is evident that by comprehensively considering the influence of the similarity metric constructed from gray-level similarity on heterogeneous image matching, solving the image optimization function constructed therefrom, and determining the optimal matching parameters through population optimization, the accuracy and robustness of heterogeneous image matching are improved, solving the problem of low accuracy in heterogeneous image matching caused by large gray-level and scale differences.

[0076] Reference Figure 4 As shown, this embodiment of the invention discloses a specific method for heterogeneous image matching. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0077] Step S21: Calculate the scale ratio of the reference image and the image to be matched based on the camera focal length and camera parameters of the reference image and the image to be matched; determine the initial matching parameters based on the scale ratio, the length and width of the reference image, and the length and width of the image to be matched.

[0078] In this embodiment, the image to be matched is first spatially transformed to obtain an initial transformed image to be matched, such as... Figure 5 As shown. Initial matching parameters are calculated using image imaging parameters and characteristics. Specifically, the scale ratio *s* between the reference image and the image to be matched is calculated. Then, based on the scale ratio, the image information of the image to be matched, and the reference image, the initial matching parameters are calculated. The formula for calculating the scale ratio is:

[0079]

[0080] Among them, f r and f s Let d represent the camera focal lengths of the reference image and the image to be matched, respectively. r and d s These represent the physical length per unit pixel calculated from the camera parameters of the reference image and the image to be matched, respectively.

[0081] The formula for calculating the initial matching parameters is as follows:

[0082]

[0083] Among them, W r H r W represents the length and width of the reference image, respectively. I H I These are the length and width of the image to be matched, respectively.

[0084] Step S22: Generate target matching parameters based on the initial matching parameters.

[0085] Step S23: Determine the overlapping image region between the reference image and the image to be matched based on the initial matching parameters.

[0086] Step S24: Based on the overlapping region, determine the reference image overlapping region in the reference image and the image to be matched overlapping region in the image to be matched.

[0087] Step S25: Construct an image optimization function using a grayscale similarity function based on the overlapping regions of the reference image and the overlapping regions of the image to be matched.

[0088] For more detailed processing procedures in steps S22, S23, S24, and S25, please refer to the aforementioned disclosed embodiments; they will not be repeated here.

[0089] Step S26: Calculate the current similarity function value of the image optimization function, and compare the current similarity function value with the target similarity function value.

[0090] In this embodiment, particle swarm initialization is performed first. Specifically, some small perturbations are added to the initial matching parameters to construct n sets of matching parameters. A population is formed by a set of matching parameters; the number of iterations is initialized to t=0; the matching parameters are obtained using the quantum particle swarm optimization algorithm.

[0091] (1) Set the maximum number of iterations MAXITER = 100, and the given error is 0.0001. (Note: The original text contains some inconsistencies and unclear formatting. A more accurate translation would require the full context.) x i (t) represents the current position of the i-th particle; P i (t) represents the current optimal position of the o-th particle; P g (t) represents the global optimal position of the particle swarm; initialize P i (0)=x i (0).

[0092] (2) Determine the initial global optimal position: Using the initial particles as matching parameters, correct the image to be matched to obtain the corrected image to be matched; gray-level statistical similarity uses normalized mutual information to calculate the similarity function value of the overlapping region of the image matching. Initial global optimal position Let I be the position of the smallest particle; where I r Represents a reference image, I s Indicates the image to be matched. This represents the corrected image to be matched. Indicates matching and merging images. This is represented by the gray-level statistical similarity function of the overlapping areas between the reference image and the corrected image to be matched. It is represented as a gray-scale structural similarity function of the overlapping regions of the reference image and the matched fused image.

[0093] (3) Update the position of each particle: Calculate the average optimal position of the population. Calculate the random location, P id (t)=φ·P id (t)+(1-φ)·P gd (t), φ = rand; Update particle position, x id (t+1)=p id (t)±α·|mbest d (t)-x id (t)|·ln(1 / u),u=rand, where,

[0094] (4) Update the current optimal position of the i-th particle: for each particle x i (t+1), using x i (t+1) is used as the matching parameter to correct the image I to be matched. s The corrected image to be matched is obtained. Calculate the similarity function value of overlapping regions in the matching images Use P i (t) is used as a matching parameter to correct the image I to be matched. s The corrected image is obtained. Calculate the similarity function value of overlapping regions in an image.

[0095] Step S27: If the current similarity function value is less than the target similarity function value, then update the current similarity function value to the target similarity function value; execute the step of calculating the current similarity function value of the image optimization function until the current iteration number meets the preset iteration number to obtain the optimal matching parameters.

[0096] In this embodiment, based on the calculations of the above process and the determination of the current optimal result for the i-th particle, if... Then P i (t+1)=x i (t+1), Otherwise, P i (t+1)=P i (t), Then update the optimal position of the group, specifically using P. g (t) is used as a matching parameter to correct the image I to be matched. s The corrected image is obtained. Calculate the similarity function value of overlapping regions in an image. if Then P g (t+1)=P i (t+1).

[0097] Set the iteration count to t = t + 1, and repeat the steps of updating the position of each particle, updating the current optimal position of the i-th particle, and updating the optimal position of the swarm until the iteration count > MAXITER, or the difference in similarity function values ​​between two consecutive optimal matches is less than the given error. Output the global optimal position P of the swarm. g (t) represents the optimal matching parameter.

[0098] Step S28: Apply the optimal matching parameters to the image to be matched using an affine transformation, and output the matched target image.

[0099] In this embodiment, the optimal matching parameters are used to perform an affine transformation on the image to be matched, and the corrected image to be matched and the matching fusion result image are output.

[0100] A set of aerial images of photovoltaic modules taken in a mountainous scene using the DJI Zenmuse XT2 dual thermal imaging camera were subjected to geometric correction and barrel correction. The relevant image acquisition parameters are shown in Table 1.

[0101] Table 1

[0102]

[0103] By combining camera parameters and performing calculations, an initial set of matching parameters was obtained as (3.7673, 0, 794.4598, 0, 3.7673, 535.5678, 21.5342). Using these parameters, the initially corrected image to be matched was obtained. The fused image obtained by fusing this image with the reference image is shown below. Figure 6 As shown, misalignment exists at the overlapping edges of the images, failing to accurately correspond. Therefore, a group of particles is first initialized in the solution space, with the QPSO particle number set to 6, indicating a solution length of 6, a maximum number of iterations of 100, a given error of 0.0001, and a swarm size of 50. The initial particle swarm is constructed using a random perturbation algorithm, and the particle similarity measure is calculated. Then, the particles search for their individual optimal positions and the global optimal positions through the QPSO algorithm, completing the optimization process after a certain number of iterations. Table 2 records the root mean square error between the initial and optimized parameters and the actual values. It can be seen that the parameter values ​​obtained from the registration experiment involving the algorithm are closest to the actual values, resulting in higher solution accuracy. The results obtained by using the optimized registration parameters in the fusion experiment are shown below. Figure 7 As shown, compared with the initial parameter fusion, the reference image and the image to be matched achieved better registration and fusion results, which visually verifies the high accuracy of the proposed method.

[0104] Table 2

[0105] Parameter type a1 b1 c1 a2 b2 c2 RMSE Actual parameters 3.6301 0.0118 838.5973 -0.0229 3.6327 561.1596 - initial parameters 3.7673 0 794.4598 0 3.7673 535.5678 20.829 Optimize parameters 3.6248 -0.0033 840.0401 -0.0325 3.6430 561.9754 0.676

[0106] Therefore, by optimizing the calculation of the similarity of the overlapping regions of the reference image and the corrected image to be matched, the problem of optimizing the matching of heterogeneous images with only partial matching regions is solved.

[0107] Reference Figure 8 As shown, this embodiment of the invention discloses a specific method for heterogeneous image matching. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically:

[0108] Step S31: Obtain the initial matching parameters of the reference image and the image to be matched, and generate the target matching parameters based on the initial matching parameters.

[0109] Step S32: Determine the overlapping image region between the reference image and the image to be matched based on the initial matching parameters.

[0110] Step S33: Based on the overlapping region, determine the reference image overlapping region in the reference image and the image to be matched overlapping region in the image to be matched.

[0111] Step S34: Construct a gray-scale statistical similarity function based on the entropy of the overlapping region of the reference image, the entropy of the overlapping region of the image to be matched after correction by the target matching parameters, and the joint entropy of the overlapping region of the reference image and the overlapping region of the image to be matched.

[0112] In this embodiment, the gray-level statistical similarity function adopts any one of mutual information, normalized mutual information, or region mutual information. When the gray-level statistical similarity function adopts normalized mutual information, the specific formula is as follows:

[0113]

[0114] Where H(·) represents the entropy of the image, It is the joint entropy of the image.

[0115] Step S35: Construct a grayscale structure similarity function based on the grayscale mean of the overlapping region of the reference image, the grayscale mean of the overlapping region of the fused image to be matched, the grayscale variance of the overlapping region of the reference image, and the grayscale variance of the overlapping region of the fused image to be matched.

[0116] In this embodiment, the specific formula for the grayscale structure similarity function is:

[0117]

[0118] Where, μ r and μ fi σ represents the mean gray level of the overlapping region between the reference image and the fused image, respectively. r and σ fi C1, C2, and C3 represent the gray-level variances of the overlapping regions of the reference image and the fused image, respectively, and are constant terms.

[0119] Step S36: Based on the target matching parameters, iteratively process the image optimization function constructed by the gray-level statistical similarity function and the gray-level structural similarity function to determine the optimal matching parameters.

[0120] Step S37: Apply the optimal matching parameters to the image to be matched using an affine transformation, and output the matched target image.

[0121] For a more detailed implementation process in steps S31, S32, S33, S36, and S36, please refer to the aforementioned disclosed embodiments; they will not be repeated here.

[0122] Therefore, by comprehensively considering the gray-level statistical characteristics, brightness, contrast, structure and other information between the original image and the matched and fused image to construct a similarity metric, the robustness of heterogeneous image matching to gray-level differences can be improved. Furthermore, by calculating the similarity of the overlapping area between the reference image and the corrected image to be matched and optimizing it, the problem of optimizing the matching of heterogeneous images with only partial matching regions can be solved.

[0123] Reference Figure 9 As shown, this embodiment of the invention also discloses a heterogeneous image matching device, comprising:

[0124] The initial parameter acquisition module 11 is used to acquire the initial matching parameters of the reference image and the image to be matched, and to generate the target matching parameters based on the initial matching parameters;

[0125] The overlapping region determination module 12 is used to determine the overlapping image region between the reference image and the image to be matched based on the initial matching parameters;

[0126] The image determination module 13 is used to determine the reference image overlapping region in the reference image and the image to be matched overlapping region in the image to be matched based on the overlapping region.

[0127] The optimization function construction module 14 is used to construct an image optimization function using a gray-level similarity function constructed based on the overlapping regions of the reference image and the overlapping regions of the image to be matched.

[0128] The optimal parameter determination module 15 is used to iteratively process the image optimization function based on the target matching parameters to determine the optimal matching parameters;

[0129] The matching image acquisition module 16 is used to perform an affine transformation on the image to be matched using the optimal matching parameters, and output the matched target image.

[0130] Therefore, this application obtains initial matching parameters for a reference image and an image to be matched, and generates target matching parameters based on the initial matching parameters; based on the initial matching parameters, it determines the overlapping image region between the reference image and the image to be matched; based on the overlapping region, it determines the reference image overlapping region in the reference image and the image to be matched overlapping region in the image to be matched; it constructs an image optimization function using a gray-level similarity function based on the reference image overlapping region and the image to be matched overlapping region; it iteratively processes the image optimization function based on the target matching parameters to determine the optimal matching parameters; and it performs an affine transformation on the image to be matched using the optimal matching parameters to output the matched target image. It is evident that by comprehensively considering the influence of the similarity metric constructed from gray-level similarity on heterogeneous image matching, solving the image optimization function constructed therefrom, and determining the optimal matching parameters through population optimization, the accuracy and robustness of heterogeneous image matching are improved, solving the problem of low accuracy in heterogeneous image matching caused by large gray-level and scale differences.

[0131] Furthermore, embodiments of this application also disclose an electronic device, Figure 10 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0132] Figure 10 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the heterogeneous image matching method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0133] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0134] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0135] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0136] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the heterogeneous image matching method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0137] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned heterogeneous image matching method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0138] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0139] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementation should not be considered beyond the scope of this application. The steps of the methods or algorithms described in connection with the embodiments disclosed herein can be implemented directly in hardware, software modules executed by a processor, or a combination of both. Software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0140] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0141] The above provides a detailed description of the heterogeneous image matching method, apparatus, device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method of heteroimage matching, characterized by, The method comprises the following steps: obtaining initial matching parameters of a reference image and a to-be-matched image, and generating target matching parameters based on the initial matching parameters; determining an overlapping image region between the reference image and the to-be-matched image based on the initial matching parameters; determining a reference image overlapping region in the reference image and a to-be-matched image overlapping region in the to-be-matched image based on the overlapping region; constructing an image optimization function by using a gray similarity function constructed based on the reference image overlapping region and the to-be-matched image overlapping region; iteratively processing the image optimization function based on the target matching parameters to determine optimal matching parameters; performing affine transformation on the to-be-matched image by using the optimal matching parameters, and outputting a target image after matching; In the process of constructing the image optimization function by using the gray similarity function constructed based on the reference image overlapping region and the to-be-matched image overlapping region, the method further comprises the following steps: constructing a gray statistical similarity function based on the entropy of the reference image overlapping region, the entropy of the to-be-matched image overlapping region after correction by the target matching parameters, and the joint entropy of the reference image overlapping region and the to-be-matched image overlapping region; In the process of constructing the image optimization function by using the gray similarity function constructed based on the reference image overlapping region and the to-be-matched image overlapping region, the method further comprises the following steps: constructing a gray structure similarity function based on the gray mean of the reference image overlapping region, the gray mean of the to-be-matched image overlapping region after fusion, the gray variance of the reference image overlapping region, and the gray variance of the to-be-matched image overlapping region after fusion; wherein the to-be-matched image overlapping region after fusion is determined based on the reference image overlapping region and the to-be-matched image overlapping region after correction.

2. The heterogenous image matching method of claim 1, wherein, The method of obtaining initial matching parameters of a reference image and a to-be-matched image comprises the following steps: calculating a scale ratio of the reference image and the to-be-matched image based on the camera focal length and the camera parameters of the reference image and the to-be-matched image; determining the initial matching parameters based on the scale ratio, the length and width of the reference image, and the length and width of the to-be-matched image.

3. The heterogenous image matching method of claim 1, wherein, The method of determining an overlapping image region between the reference image and the to-be-matched image based on the initial matching parameters comprises the following steps: determining target coordinates in the reference image corresponding to the to-be-matched image by using a preset coordinate transformation formula and the vertex coordinates of the to-be-matched image; performing integer sorting operations on the horizontal coordinates and the vertical coordinates of the target coordinates, respectively, to determine position information of the overlapping image region; determining the overlapping image region between the reference image and the to-be-matched image based on the position information.

4. The heterogenous image matching method of claim 1, wherein, The method of iteratively processing the image optimization function based on the target matching parameters to determine optimal matching parameters comprises the following steps: calculating a current similarity function value of the image optimization function, and comparing the current similarity function value with a target similarity function value; if the current similarity function value is smaller than the target similarity function value, updating the current similarity function value to the target similarity function value; The step of performing calculation on the current similarity function value of the image optimization function is executed until the preset iteration number is met, so as to obtain the optimal matching parameter.

5. A device for matching of heterogenous images, characterized in that The method comprises the steps of: An initial parameter acquisition module is configured to acquire initial matching parameters of a reference image and a to-be-matched image, and generate target matching parameters based on the initial matching parameters. An overlapping area determination module is configured to determine an overlapping image area between the reference image and the to-be-matched image based on the initial matching parameters. An image determination module is configured to determine a reference image overlapping area in the reference image and a to-be-matched image overlapping area in the to-be-matched image based on the overlapping area. An optimization function construction module is configured to construct an image optimization function by using a gray-scale similarity function constructed based on the reference image overlapping area and the to-be-matched image overlapping area. An optimal parameter determination module is configured to perform iterative processing on the image optimization function based on the target matching parameters, so as to determine an optimal matching parameter. A matched image acquisition module is configured to perform affine transformation on the to-be-matched image by using the optimal matching parameter, and output a target image after matching. The heterogeneous image matching device is further configured to construct a gray-scale statistical similarity function based on an entropy of the reference image overlapping area, an entropy of the to-be-matched image overlapping area after correction by the target matching parameter, and a joint entropy of the reference image overlapping area and the to-be-matched image overlapping area. The heterogeneous image matching device is further configured to construct a gray-scale structural similarity function based on a gray-scale mean value of the reference image overlapping area, a gray-scale mean value of the to-be-matched image overlapping area after fusion, a gray-scale variance of the reference image overlapping area, and a gray-scale variance of the to-be-matched image overlapping area after fusion.

6. An electronic device, comprising: The method comprises the steps of: A memory is configured to save a computer program. A processor is configured to execute the computer program, so as to implement the steps of the heterogeneous image matching method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer program is stored in the memory. When the computer program is executed by the processor, the steps of the heterogeneous image matching method according to any one of claims 1 to 4 are implemented.

Images