Spatial Target Heterogeneous Image Matching Method Based on Image Completion

Abstract

A method for matching heterogeneous images of space targets based on image completion belongs to the technical fields of ISAR image and optical image processing. In order to solve the problem of low matching accuracy existing in the current matching image method for the case of missing ISAR pixels. First, the present invention simulates an ISAR image sequence in the case of target spin based on a three-dimensional space target model and a range-Doppler algorithm; and obtains optical samples of the space target in the same pose as the ISAR image by using a projection method; then, based on the image sequence and the HaLRTC method with optimal weights, the missing pixels of the ISAR image corresponding to the space target are completed; finally, based on the method of pyramid resampling and optimized iteration, the ISAR image and the optical image of the space target are matched. The present invention is mainly used for the matching of ISAR images and optical images.

Description

Technical Field

[0001] The present invention relates to a method for matching heterogeneous images of space targets, belonging to the technical fields of ISAR image and optical image processing. Background Art

[0002] The matching technology of heterogeneous visual images is a key technology widely used in visual navigation, pattern recognition, guidance, and topographic survey of aircraft, remote sensing satellites, missiles, etc. These systems often use sensors with different imaging mechanisms. The so-called heterogeneous image matching is a technology for matching images from different imaging sensors. These images are images of the same scene, object, or target formed by different imaging sensors under different lighting environments and other conditions and different imaging mechanisms, mainly including visible light images, infrared images, and radar images (ISAR images), etc. Due to the differences in the structures, imaging principles, etc. of different types of sensors themselves, there are large differences in the gray levels and contrasts of the corresponding regions on the heterogeneous images. Therefore, heterogeneous image matching is a very difficult task.

[0003] For the ISAR images of space targets, due to the influence of space electromagnetic interference, the ISAR echo data often faces the problem of missing data, which also leads to missing pixel points in the presented ISAR images. Therefore, before performing heterogeneous image matching, it is necessary to first complete the image pixel points. At present, many domestic and foreign literatures introduce algorithms for directly matching heterogeneous images. Due to the missing ISAR pixel points, and even these missing points may exist in key matching positions, such as on the solar wings for satellite images, this results in low matching accuracy when matching images, which is also the technical problem to be solved by the present invention. Summary of the Invention

[0004] The present invention is to solve the problem of low matching accuracy in the current method for matching images in the case of missing ISAR pixel points.

[0005] A method for matching heterogeneous images of space targets based on image completion includes the following steps:

[0006] Step 1: Simulate an ISAR image sequence in the case of target spin based on a three-dimensional space target model and the range-Doppler algorithm; and obtain an optical sample of the space target with the same pose as the ISAR image by using the projection method;

[0007] Step 2: Complete the missing pixels of the ISAR image corresponding to the space target based on the image sequence and the HaLRTC method with optimal weights;

[0008] Step 3: Match the ISAR image and the optical image of the space target based on the method of pyramid resampling and optimization iteration.

[0009] Furthermore, the process of using the HaLRTC method based on the image sequence and optimal weights to complete the missing pixels of the spatial target ISAR image in step 2 includes the following steps:

[0010] 2.1. Construct an ISAR image sequence containing the spatial target, use the ISAR image containing the spatial target as a tensor X, and divide the tensor into three input channel tensors X i , i = 1, 2, 3, X i with a size of n×n×N;

[0011] 2.2. Use the missing model to construct the ISAR image missing pixel sequence. For each input channel tensor, there is a missing model S i , i = 1, 2, 3. After being processed by the missing model, the ISAR image missing pixel sequence is represented as a tensor with a size of n×n×N;

[0012] 2.3. Use the HaLRTC method based on optimized weights for image completion:

[0013] 2.3.1. For K ISAR image sequences, construct the ISAR image missing pixel sequence

[0014] 2.3.2. Calculate the loss function and update the weights:

[0015]

[0016]

[0017]

[0018] where k represents the k-th iteration, is the weight corresponding to the i-th channel of the j-th image sequence at the (k + 1)-th round, represents the loss function; lr is the update rate;

[0019] Record the current weights in each iteration Discard the two largest and two smallest weight values in each channel, and obtain the optimized weights using the following formula:

[0020]

[0021] When the loss function is less than the threshold L gate stop the iteration, or stop the iteration when the maximum number of iterations is reached, and substitute the obtained optimized weights into the HaLRTC method for image completion to obtain the completed tensor data for each channel Combine the data of each channel in sequence to obtain the complemented ISAR image data.

[0022] Further, the process of matching the spatial target ISAR image and the optical image by the method based on pyramid resampling and optimization iteration described in step 3 includes the following steps:

[0023] 3.1. Filter the complemented ISAR image to obtain the filtered ISAR image W;

[0024] Then convert both the optical image and the ISAR image into grayscale images;

[0025] 3.2. Set a pre-parameter w c , and perform affine transformations such as translation, rotation, and scaling on the optical image; the optical image O after affine transformation is

[0026] The affine transformation matrix is S′, and the pre-parameter w c is the parameter initially set for the affine transformation matrix;

[0027] 3.3. Construct Laplacian pyramid residual images for the filtered image W corresponding to the ISAR image converted into a grayscale image and the affine-transformed optical image respectively;

[0028] 3.4. Use the gradient descent optimizer to optimize the matching result:

[0029]

[0030] where w old represents the parameter before the update of the pre-parameter w c , and w new represents the parameter after the update of the pre-parameter w c , is the gradient of the loss value with respect to the parameter, which determines the direction of parameter update, and η is the learning rate; the loss value loss(·) is the mutual information MI(A, B) between the filtered image W and the Laplacian pyramid residual images corresponding to the optical image A and B are the Laplacian pyramid residual images corresponding to the filtered image W and the optical image respectively;

[0031] When the mutual information MI(A, B) is less than the mutual information threshold Γ, continue the iteration until the mutual information threshold Γ is reached;

[0032] 3.5. Use the method of diffeomorphism for non-rigid matching.

[0033] Further, the filtered image W and the optical image The mutual information MI(A, B) of the corresponding Laplacian pyramid residual images is as follows

[0034] MI(A, B) = H(A) + H(B) - H(AB) (8)

[0035] Where

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042] Where A and B are the filtered image W and the optical image respectively The corresponding Laplacian pyramid residual images; a and b are the pixel points corresponding to the images respectively; h(a, b) is the joint gray-level histogram between the images.

[0043] Furthermore, the process of constructing the Laplacian pyramid residual images described in step 3.3 includes the following steps:

[0044] First, construct the Gaussian pyramid:

[0045]

[0046] Where G l-1 (·) represents the l - 1 level pyramid image; l represents the level of the image, q and e are two variables; w(q, e) is the Gaussian template;

[0047] Then perform the reverse operation. Through the interpolation operation, expand an image at a certain level in the Gaussian pyramid to the size of the previous level image, which is expressed as follows:

[0048]

[0049] Subtract the original image of the previous level from the result of interpolating the image of the previous level to obtain the image of this level, that is, the residual image, G l+1 (i, j) is the expression of the l + 1 level, which subtracts the image of the previous level to obtain the Laplacian pyramid residual image.

[0050] Further, before filtering the completed ISAR image and before converting the optical image into a grayscale image, it is necessary to filter the ISAR image and scale the optical image according to the markers in the images so that the spatial proportion difference of the heterogeneous images does not exceed 10%.

[0051] Further, the process of filtering the completed ISAR image is implemented by enhanced Lee filtering.

[0052]

[0053] Where I represents the image pixels within the sliding window. is the mean value of the pixels within the sliding window, W is the estimation of the image, that is, the filtered image; C I represents the local standard deviation coefficient of the image, C min = C u , L represents the equivalent number of looks, u represents the noise, C u is the local standard deviation coefficient of the noise; p is the weighting coefficient of the Lee filtering.

[0054] Further, the process of constructing the ISAR image sequence containing the spatial target described in step 2.1 includes the following steps:

[0055] Select a continuous sequence of N ISAR images, intercept a square region with a size of n×n at the center of the image, ensure that this region can completely contain the spatial target, and after interception, form the input data tensor X with a size of n×n×3×N, where 3 represents the three color channels of the ISAR image.

[0056] Further, the process of simulating the ISAR image sequence in the case of target spin based on the three-dimensional spatial target model and the range-Doppler algorithm includes the following steps:

[0057] 1.1. First, open the 3D model file of the spatial target, and save it as a parasolid format file and an mdl file convenient for importing into FEKO software.

[0058] 1.2. Open the.x_t file in FEKO software, set the solution algorithm, target material, mesh quality, and electromagnetic wave carrier frequency parameters, and after establishing the solid mesh of the target, save it as a.nas framework file.

[0059] 1.3. After saving the framework file as a text file, process the strings in the text file, export the grid point coordinates of the scatter point group to MATLAB, uniformly select N point coordinates from the output results, and make the target completely displayed on the ISAR image, so as to obtain the discrete scatter point distribution of the model.

[0060] 1.4. Perform ISAR imaging on the scattering points with known distributions using the range-Doppler algorithm, and simulate the ISAR image sequence in the case of target spin.

[0061] Further, the construction process of the missing model includes the following steps:

[0062] Assume that the size of the ISAR image X pixel is m×m×3, and some of its elements are missing. The observed index set is (i, j, k) ∈ Ω, where i, j, and k are the indices of the ISAR image pixels on the three channels respectively. Ω is the observation set, and the pixels not in this set are the missing pixels. Let the missing model S satisfy s ij = 1, (i, j, k) ∈ Ω, otherwise Then the ISAR image after missing pixels can be expressed as

[0063] Beneficial effects:

[0064] The method of the present invention is different from the traditional heterogeneous image matching method. It uses a two-step strategy. First, it is concerned that a large part of the reason for the low matching accuracy between the ISAR image and the optical image is the problem of missing pixel points in the imaging process of the ISAR image. For this problem, the image pixel points are first complemented, and an innovative HaLRTC (high accuracy low rank tensor completion) method based on optimized weights is used; in the second step, the heterogeneous matching of the complemented image is performed using the pyramid resampling and optimized iteration method. This optimized iteration method has greatly improved the accuracy compared to the traditional method of extracting feature points for registration.

[0065] The present invention generally includes three parts: space target ISAR and optical samples, image completion using the HaLRTC method based on sequences and optimized weights, and heterogeneous image matching based on pyramid resampling and optimized iteration; the space target ISAR and optical samples part is responsible for simulating and generating a large number of ISAR images in the spin state of space targets and optical image sequences in the same pose; the image completion part using the HaLRTC method based on sequences and optimized weights obtains the pixel distribution by inputting the ISAR image sequence samples of a certain channel, then uses the iterative method to obtain the optimal weight, and then substitutes this optimal weight into the original algorithm to achieve image completion; in the final matching part, the optical image is first rotated, translated, and scaled to ensure image mismatch, and then the pyramid resampling and optimized iteration method is used to match the complemented ISAR image and the corresponding optical image. Finally, the differential homeomorphism non-rigid matching method is used to optimize the result.

[0066] The heterologous image matching method adopting the two-step strategy of the present invention can match the ISAR image with missing pixel points in the actual situation with the optical image. The optimized weight HaLRTC method after training can obtain the optimal weight that enables the algorithm to converge quickly and has a good complementation effect, improving the accuracy of the subsequent matching result, meeting the actual needs, and being convenient to implement.

[0067] The present invention obtains the scattering point information required for ISAR imaging through the three-dimensional model and electromagnetic parameter simulation of the space target. The ISAR imaging result is almost the same as the measured ISAR image, and the ISAR simulation image sequence and optical image sequence of five types of targets are obtained, ensuring the demand for samples of the method in this paper and the smooth application of the algorithm.

[0068] The present invention innovatively proposes the HaLRTC method based on optimized weights to complement pixels of the ISAR image, and the method based on pyramid resampling and optimized iteration to match heterologous images. The input is the ISAR space target image sequence and optical image sequence with missing pixel points, and the output is the affine transformation matrix from the optical image to the ISAR image, having certain method superiority.

[0069] The present invention utilizes ISAR imaging technology and digital image processing technology to better meet the demand for heterologous image matching in different scenarios. This method maintains a certain degree of stability for the perspective change, affine transformation, and noise of the input image. Description of the Drawings

[0070] Figure 1 is the overall flowchart of the heterologous image matching method for space targets based on image completion;

[0071] Figure 2 is the schematic flowchart of image completion;

[0072] Figure 3 is the flowchart of image matching based on the method of pyramid resampling and optimized iteration;

[0073] Figure 4 is the flowchart for obtaining the simulation ISAR image and optical image sequence;

[0074] Figure 5(a) is the original ISAR image, Figure 5(b) is the image with missing pixels, and Figure 5(c) is the image after completion;

[0075] Figure 6(a) is the registered optical image and ISAR image of the space target, and Figure 6(b) is the registered optical image and ISAR of the space target. Detailed Implementation Manner

[0076] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0077] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other. The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but it is not limited to the present invention.

[0078] Specific Embodiment 1: Combined with Figures 1 to 4 Describe this embodiment

[0079] A method for matching heterogeneous images of space targets based on image completion includes the following steps:

[0080] Step 1: Obtain a sequence of ISAR spin space target samples based on a three-dimensional space target model and the range-Doppler algorithm simulation; and obtain optical samples of the space target with the same pose as the ISAR image by using the projection method.

[0081] Step 2: Complete the missing pixels of the space target ISAR image based on the image sequence and the optimal weight HaLRTC method.

[0082] Step 3: Match the space target ISAR image and the optical image based on the pyramid resampling and optimization iteration method.

[0083] The specific process of the above Step 1 includes the following steps:

[0084] 1.1. First, open the 3D model file of the space target in SOLIDWORKS software, and save it as a parasolid format file and an mdl file convenient for importing into FEKO software.

[0085] The 3D model in this embodiment is obtained by inverting the target through 3Dmax. Of course, a ready-made model downloaded from the Internet can also be used.

[0086] 1.2. Open the.x_t file in FEKO software, set parameters such as the solution algorithm, target material, mesh quality, and electromagnetic wave carrier frequency, and save it as a.nas framework file after establishing the solid mesh of the target.

[0087] 1.3. Save the frame file as a text file and process the strings in the text file. Export the grid point coordinates of the scattering point group to MATLAB, and uniformly select N point coordinates from the output results. Multiply by a coefficient to make the complete spatial target appear in the ISAR image, thereby obtaining the discrete scattering point distribution of the model.

[0088] The specific value of the coefficient depends on the electromagnetic wave frequency used in the simulation.

[0089] 1.4. Use the range-Doppler algorithm to perform ISAR imaging on the scattering points with known distribution, and simulate the ISAR image sequence under the condition of target spin, that is, the ISAR spin spatial target sequence; and obtain the optical image O with the same pose as the ISAR image based on the mdl model in step 1.1 using the projection method.

[0090] Before performing step 2, a missing model needs to be constructed. The process of constructing the missing model includes the following steps:

[0091] Assume that the size of the ISAR image X pixel is m×m×3, and some of its elements are missing. The observed index set is (i,j,k)∈Ω, where i, j, and k are the indices of the image pixels on the three channels respectively, and Ω is the observation set. Those not in this set are the missing pixels; let the missing model S satisfy s ij = 1, (i,j,k)∈Ω, otherwise Then the ISAR image after missing pixels can be expressed as

[0092] s ij is an element of the missing model. The missing model is equivalent to a template, consisting of 1s and 0s. Each channel data of the image is multiplied by a different template, which is the result of pixel missing.

[0093] The process of using the HaLRTC method based on the image sequence and optimal weight to complete the missing pixels of the spatial target ISAR image in step 2 includes the following steps:

[0094] 2.1. Since the principle of using the HaLRTC method to complete the ISAR image pixels is to utilize the correlation of sample pixels between similar ISAR images, an ISAR image sequence needs to be established first. Construct the ISAR image sequence:

[0095] Select a continuous sequence of N ISAR images, where the size of N should be less than the maximum value N max, ensure that the target rotation angle and attitude change in the sequence are within a certain range; intercept a square area of size n×n at the center of the image to ensure that this area can completely contain the spatial target. After interception, form the input data tensor X, with a size of n×n×3×N, where 3 represents the three color channels of the ISAR image. Divide the input data tensor into three input channel tensors X i , i = 1, 2, 3, X i has a size of n×n×N;

[0096] Actually, the simulated image sequence in step 1 is the overall data set. In this embodiment, there are 200 images of each type of target in the simulated image sequence in step 1, while in step 2.1, there may be 10 images of each type in the selected sequence. In order to use the HaLRTC method for image completion, the target images need to exist in the form of a continuously changing sequence. Essentially, it is to use the pixel relationship between images for completion, which can be understood as the relationship between the total training set and the training batch.

[0097] 2.2. Construct the missing pixel sequence of the ISAR image:

[0098] Simulate the data missing situation in the actual ISAR imaging process, and use the missing model to construct the missing pixel sequence of the ISAR image. For each input channel tensor, there is a missing model S i , i = 1, 2, 3. After being processed by the missing model, the missing pixel sequence of the ISAR image can be represented as a tensor with a size of n×n×N;

[0099] 2.3. Use the HaLRTC method based on optimized weights for image completion:

[0100] Since the convergence speed of the HaLRTC method for image completion depends to a large extent on the weight ρ it selects, the present invention proposes a HaLRTC method based on optimized weights.

[0101] 2.3.1. First, select K ISAR image sequences, and construct the missing pixel sequence of the ISAR image according to steps 2.1 - 2.2

[0102] 2.3.2. Set the hyperparameters in HaLRTC;

[0103] The hyperparameters include:

[0104] For the HaLRTC method based on optimized weights, the number of ISAR image sequences is selected as 10, the number of training samples K is selected as 20, the maximum number of iterations maxiter is 1000, and the initial weight ρ init is 0.01, and the update rate lr is 0.0001;

[0105] For the gradient descent optimizer, the number of spatial samplings is 500, the number of histogram bins is 50, the growth factor is 1.05, the initial search radius is 0.000625, the maximum number of iterations is 200, and the pyramid is set to 3 layers;

[0106] For the diffeomorphic non-rigid matching method, the number of iterations per pyramid layer is set to 100, the smoothness is set to 1.5, and the pyramid is set to 7 layers;

[0107] The update of the weights and the loss function are set using the following formula:

[0108]

[0109]

[0110]

[0111] where k represents the k-th iteration, is the weight corresponding to the i-th channel of the j-th image sequence at the (k + 1)-th iteration, L represents the loss function; lr is the update rate.

[0112] Record the current weights in each iteration Discard the two largest and two smallest weight values in each channel, and obtain the optimized weights using the following formula:

[0113]

[0114] According to the loss function in formula (1), when it is less than the threshold L gate stop the iteration, or stop the iteration when the maximum number of iterations is reached, and substitute the obtained optimized weights into the HaLRTC method for image completion to obtain the tensor data of each channel after completion Combine the data of each channel in order to obtain the ISAR image data after completion;

[0115] In this embodiment, the pixel sizes of the simulated spatial target ISAR image and the optical image are both 256×256×3; the size of the sequence tensor obtained by constructing the ISAR image sequence is 256×256×3×200; the size of the intercepted input data tensor is 128×128×3×200; for the HaLRTC method based on optimized weights, each input channel tensor is 128×128×10, and there are 20×3 groups of input channel tensors in total. The size of the tensor after being processed by the missing model is 128×128×10. 20 ISAR missing pixel sequences are selected for training in each channel, and the optimal weights of each channel are obtained through 1000 iterations by the method in step 2.3.2 Substitute it into the HaLRTC method and set the threshold Lgate With a size of 1000, the iteration stops when the maximum iteration number or threshold is reached, and the completed image data is obtained. Each channel of each image has a size of 128×128. Combining the three channels gives the completed image with a size of 128×128×3.

[0116] The process of matching the spatial target ISAR image and the optical image by the method based on pyramid resampling and optimization iteration described in step 3 includes the steps of image preprocessing, affine transformation of the optical image, pyramid resampling, optimizer adjustment of variable parameters, and mutual information calculation.

[0117] 3.1. Image preprocessing:

[0118] Since this method is not suitable for the registration of heterologous images with too large spatial scale differences, the images are first preprocessed. The images are scaled according to the markers in the images so that the spatial scale differences of the heterologous images do not exceed 10%. Due to the imaging mechanism problem, there is a large amount of speckle noise in the ISAR image, which has a negative impact on the registration of heterologous images. Therefore, the enhanced Lee filtering method is used to filter the completed ISAR image to remove the speckle noise in the ISAR image. In this embodiment, a 3×3 sliding window is selected for Lee filtering; and both the optical image and the ISAR image are converted into grayscale images.

[0119] The enhanced Lee filtering is performed using the following formula:

[0120]

[0121] where I represents the image pixels within the sliding window, is the mean value of the pixels within the sliding window, W is the estimation of the image, i.e., the filtered image; C I represents the local standard deviation coefficient of the image, C min =C u , L represents the equivalent number of looks, u represents the noise, C u is the local standard deviation coefficient of the noise; p is the weighting coefficient of the Lee filtering.

[0122] Specifically, the algorithm divides the image region into three categories, and the processing method for each category is different:

[0123] 1) When C I < C u (u represents the noise) indicating a uniform region, the average value of the pixels in the sliding window is used as the central value (window center value);

[0124] 2) When C u < C I < C maxWhen it represents a weak texture area, the traditional Lee filter is used to process it;

[0125] 3) When C I > C max , it represents a non-uniform area where the speckle noise is not fully developed. No other processing is performed, and the original value is directly retained.

[0126] 3.2. Due to the scale differences in remote sensing images, a pre-parameter w c is first set, and affine transformations such as translation, rotation, and scaling are performed on the optical image. Assuming the affine transformation matrix is S′, its expression is:

[0127]

[0128] The pre-parameter w c is the parameter initially set for this affine transformation matrix, including the rotation angle θ, the scaling ratio s, and the translation amounts t x and t y , t x , t y are the translation amounts in two directions in the two-dimensional translation respectively;

[0129] If the optical image is O, then the optical image after affine transformation is

[0130] 3.3. For the filtered image W corresponding to the ISAR image converted to a grayscale image and the optical image after affine transformation Laplacian pyramid residual images are respectively constructed. Since the pixel differences between heterologous images are large and the image matching ability after Gaussian pyramid downsampling is not strong, Laplacian pyramid residual images are adopted in the present invention. First, a Gaussian pyramid is constructed using formula (4):

[0131]

[0132] where, G l-1 (·) represents the l - 1 layer pyramid image; l represents the layer of the image, q and e are two variables; w(q,e) is the Gaussian template. This formula means multiplying the source image with the Gaussian template and then adding, and then performing interval sampling;

[0133] Then, using the reverse operation of the above formula, through interpolation operation, usually inserting 0 points, an image at a certain level in the Gaussian pyramid is expanded to the size of the previous level image, which is expressed as follows:

[0134]

[0135] The Laplacian pyramid, i.e., the Laplacian pyramid residual image, is obtained through many operations of interpolation and subtraction. The Laplacian pyramid residual image is actually the result of interpolating the image of the previous layer and then subtracting the original image of the previous layer to obtain the image of this layer, which is the residual image, G l+1 (i,j) is not the final result of the residual image. It is the expression of the (l + 1)-th layer. Subtracting the image of the previous layer from it gives the residual image.

[0136] 3.4. Optimize the matching result using the gradient descent optimizer. The algorithm can be expressed as:

[0137]

[0138] where w old represents the pre-parameter w c the parameter before update, w new represents the pre-parameter w c the parameter after update, is the gradient of the loss value with respect to the parameter, which determines the direction of parameter update. The loss value loss(·) is the mutual information MI(A,B); η is the learning rate, which scales the length of the gradient advancement and determines the speed of parameter update.

[0139] To describe this process more clearly, the above process can be decomposed into three steps. g represents the gradient, v is the learning rate η multiplied by the negative gradient -g, and the update amount of the parameter vector:

[0140]

[0141] The conventional step-size gradient descent optimizes and adjusts the transformation parameters to make the optimization follow the gradient of the image similarity metric in the extreme value direction. It uses a constant step size along the calculated gradient until the gradient changes direction. Then the step size gradually decreases according to the set rule until the algorithm stops iterating. The above loss value, i.e., the image similarity metric, is calculated using the mutual information of the image and is defined as:

[0142] MI(A,B) = H(A) + H(B) - H(AB) (8)

[0143] where

[0144]

[0145]

[0146]

[0147] where A and B are the input images of the algorithm, which are the filtered image W and the optical image respectively The corresponding Laplacian pyramid residual image; a and b are the corresponding pixel points of the image respectively;

[0148] The probability distribution is calculated through the joint gray-level histogram h(a, b) between images, that is:

[0149]

[0150]

[0151]

[0152] For the present invention, first, a mutual information threshold Γ that is considered to be well registered is set. When the mutual information is less than this threshold, the algorithm continues to iterate until the threshold is reached.

[0153] The "pre-parameters" in step 3.2 are initial values. The mutual information calculated with these initial values is certainly very small. Then, (6) is used to iteratively optimize until a very large mutual information, which is the threshold, is reached. When the threshold is reached, the algorithm considers that the matching is successful and the iteration is completed. In fact, HaLRTC is responsible for filling in the pixels of the ISAR image to make the subsequent matching effect better. The matching itself is an optimization process because there are rotation, translation, and scale transformations between the optical image at the beginning and the ISAR image, which can be described by an affine transformation matrix. There are four parameters in the matrix. When these parameters are optimized accurately, a good matching effect can be obtained.

[0154] 3.5. Finally, the method of diffeomorphism is introduced for non-rigid matching to make the local registration result more accurate. Hyperparameters such as smoothness and the number of pyramid layers are set to obtain a more accurate final registration result, realizing the registration of heterologous image data of space target ISAR and optical images.

[0155] In this embodiment, for the heterologous image matching method, the pixel sizes of the input ISAR image and the optical image are both 128×128×3. The ISAR image is preprocessed by enhanced Lee filtering and converted into a grayscale image together with the optical image, with a pixel size of 128×128; then, an affine transformation is performed on the optical image. First, an initial transformation matrix S′ is set, and the pixel size after transformation is 128×128; the iteration starts. First, three-layer Laplacian residual pyramids of the ISAR image and the optical image are respectively constructed, and then the loss function is obtained by calculating the mutual information using formula (8). Finally, formula (6) is used to optimize the matching result, that is, to adjust the parameters of the affine transformation matrix so that the optimization follows the gradient of the image similarity metric in the extreme value direction. It uses a constant step size along the gradient between calculations until the gradient changes direction. After that, the step size gradually decreases according to the set rules until the algorithm reaches the maximum number of iterations;

[0156] Finally, using the diffeomorphic non-rigid matching method and the hyperparameters set in claim 3, the matching result is optimized to obtain the final matching transformation matrix, realizing the matching between heterologous images of spatial targets.

[0157] Embodiment

[0158] As Figure 1 shown, this embodiment mainly includes the following steps:

[0159] First, based on the three-dimensional spatial target model and the range-Doppler algorithm, simulate to obtain the ISAR spin spatial target sequence samples; and use the projection method to obtain the optical samples of the spatial target with the same pose as the ISAR image.

[0160] Construct the ISAR spatial target image sequence with missing pixel points through the missing model, and use the HaLRTC method based on the image sequence and optimized weights to complete the missing pixels of the spatial target ISAR image; in this process, first use a part of the training sequence to obtain the optimal weights, and then use these optimal weights to complete the missing pixels of the target image sequence.

[0161] Finally, based on the pyramid resampling and optimized iteration method, match the completed spatial target ISAR image and its corresponding optical image.

[0162] In this embodiment, the ISAR spin spatial target samples are the simulated image sequences, which are used as the fixed reference images after completion. In the matching stage, the affine transformation matrix of the optical image relative to the reference image is finally obtained.

[0163] As Figure 4 shown, the process of obtaining the ISAR spin spatial samples includes: electromagnetic parameter setting and ISAR imaging. The electromagnetic parameter setting generates the grid frequency according to the size of the model, and the corner points of the grid lines on the target model are used as the spatial coordinates of the three-dimensional scattering points; in the ISAR imaging part, using the three-dimensional information of the target's scattering points, according to the set radar parameters and target motion parameters, perform ISAR imaging simulation to obtain the ISAR spin spatial target image; then simulate to obtain the images of 5 types of targets to form the simulation image sequence.

[0164] The pixel sizes of the simulated ISAR images and optical images of the space targets are both 256×256×3; the size of the sequence tensor obtained by constructing the ISAR image sequence is 256×256×3×200; the size of the input data tensor after cropping is 128×128×3×200; for the HaLRTC method based on optimized weights, each input channel tensor is 128×128×10, and there are 20×3 groups of input channel tensors in total. The size of the tensor after being processed by the missing model is 128×128×10, and 20 ISAR missing pixel sequences are selected for training in each channel; for the heterogeneous image matching method, the pixel sizes of the input ISAR image and optical image are both 128×128×3, the pixel size of the preprocessed image is 128×128; the pixel size after affine transformation is 128×128; the size of the finally obtained affine transformation matrix is 3×3;

[0165] The hyperparameters include: the grid electromagnetic wave carrier frequency set in FEKO is 1 GHz, the number of scattered points selected during the imaging process is 500, the initial angle between the reference coordinate system and the local coordinate system is set to -130°. The frequency of the chirp signal is set to f c = 5.52 GHz, the bandwidth B = 300 MHz, the pulse signal duration T p = 25.6×10 -6 s, the range sampling frequency F M = 2.5e 6 Hz. The rotational angular velocity of the space target is set to 0.1 rad / s, and there are a total of 5 types of space targets; the pixel sizes of the simulated ISAR images and optical images of the space targets are both 256×256, and each target has 200 sequences. When constructing the ISAR image sequence, n is taken as 128;

[0166] For the HaLRTC method based on optimized weights, the number of ISAR image sequences is selected as 10, the number of training samples K is selected as 20, the maximum number of iterations maxiter is 1000, and the initial weight ρ init is 0.01, and the update rate lr is 0.0001;

[0167] For the gradient descent optimizer, the number of spatial samplings is 500, the number of histogram bins is 50, the growth factor is 1.05, the initial search radius is 0.000625, the maximum number of iterations is 200, and the pyramid is set to 3 layers;

[0168] For the diffeomorphic non-rigid matching method, the number of iterations for each layer of the pyramid is set to 100, the smoothness is set to 1.5, and the pyramid is set to 7 layers;

[0169] During the process of constructing the ISAR spatial target image sequence of missing pixels through the missing model and using the HaLRTC method based on the image sequence and optimized weights to complete the missing pixels of the spatial target ISAR image, first use a part of the training sequence to obtain the optimal weights, and then use these optimal weights to complete the missing pixels of the target image sequence; finally, based on the pyramid resampling and optimized iteration method, match the completed spatial target ISAR image with its corresponding optical image; after obtaining the affine transformation matrix corresponding to the optical image, the method runs to completion. As Figure 2 shown, the operation process of obtaining the optimal weights is as follows:

[0170] First, construct an ISAR image sequence of K missing pixels. In this method, K is taken as 20, and combined into a channel input tensor with a size of 128×128×10. Then set the initial hyperparameters, including the maximum number of iterations maxiter as 1000, the initial weight ρ init as 0.01, the update rate lr as 0.0001, etc.; complete the filling through the HaLRTC method, calculate the loss during each iteration, update the weights through the calculation results, and finally obtain the optimal weights when reaching the maximum number of iterations; finally, use the optimal weights to complete the target ISAR image sequence of missing pixels, and both the speed and accuracy are improved when using the optimal weights to complete;

[0171] As Figure 3 shown in the architecture diagram of heterologous image matching, the process of heterologous image matching includes the following steps:

[0172] The pixel sizes of the input ISAR image and the optical image are both 128×128×3. The ISAR image is preprocessed by enhanced Lee filtering and converted into a grayscale image together with the optical image, with a pixel size of 128×128; then perform an affine transformation on the optical image. First, set an initial transformation matrix S, and the pixel size after transformation is 128×128; start the iteration. First, construct the three-layer Laplacian pyramid residual images of the ISAR image and the optical image, then calculate the mutual information to obtain the loss function, and finally optimize the matching result and adjust the parameters of the affine transformation matrix so that the optimization follows the gradient of the image similarity metric in the extreme value direction. It uses a constant-length step size along the calculated gradient until the gradient changes direction. Then the step size gradually decreases according to the set rules until the algorithm reaches the maximum number of iterations.

[0173] Finally, use the diffeomorphic non-rigid matching method to optimize the matching result through the set hyperparameters to obtain the final matching transformation matrix and realize the matching between the heterologous images of the spatial target.

[0174] For the effects of this embodiment, see Figures 5(a) to 5(c) , and Figures 6(a) to 6(b) .

[0175] Although the present invention has been described herein with reference to particular embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should thus be understood that numerous modifications may be made to the exemplary embodiments, and other arrangements may be devised, without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the different dependent claims and the features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in connection with a separate embodiment may be used in other described embodiments.

Claims

1. A method for matching heterogeneous images of space targets based on image completion, characterized in that, Including the following steps: Step 1: Based on the three-dimensional spatial target model and the range-Doppler algorithm, simulate to obtain the ISAR image sequence under the condition of target spin; and use the projection method to obtain the optical sample of the spatial target with the same pose as the ISAR image; Step 2: Based on the image sequence and the HaLRTC method with optimal weights, complete the missing pixels of the ISAR image corresponding to the spatial target, including the following steps: 2.

1. Construct an ISAR image sequence containing a spatial target, use the ISAR image containing the spatial target as a tensor X, and divide the tensor into three input channel tensors X i , i = 1, 2, 3, X i with the size of n×n×N; 2.

2. Using the missing model to construct the missing pixel sequence of the ISAR image, and there is a missing model S for each input channel tensor i , i = 1, 2, 3. After being processed by the missing model, the missing pixel sequence of the ISAR image is represented as a tensor with a size of n×n×N; 2.

3. Use the HaLRTC method based on optimized weights to complete the image: 2.3.

1. For K ISAR image sequences, construct the ISAR image missing pixel sequences 2.3.

2. Calculate the loss function and update the weights: Among them, $k$ represents the $k$-th iteration round, is the weight corresponding to the $i$-th channel of the $j$-th image sequence at the $(k + 1)$-th round, represents the loss function; $lr$ is the update rate; Record the current weights in each iteration Discard the two largest and two smallest weight values in each channel, and obtain the optimized weights using the following formula: When the loss function shown is less than the threshold L gate stop the iteration, or stop the iteration when the maximum number of iterations is reached, and substitute the obtained optimized weights into the HaLRTC method for image completion to obtain the tensor data of each channel after completion Combine the data of each channel in order to obtain the ISAR image data after completion; Step 3: Match the ISAR image and the optical image of the spatial target based on the pyramid resampling and optimized iteration method, including the following steps: 3.

1. Filter the completed ISAR image to obtain the filtered ISAR image W; Then convert both the optical image and the ISAR image into grayscale images; 3.

2. Set a pre-parameter w c , and perform affine transformations such as translation, rotation, and scaling on the optical image; the optical image after the affine transformation of the optical image O is The affine transformation matrix is S′, and the pre-parameter w c is the parameter initially set for the affine transformation matrix; 3.

3. For the filtered image W corresponding to the ISAR image converted into a grayscale image and the optical image after affine transformation Construct Laplacian pyramid residual images respectively; 3.

4. Use the gradient descent optimizer to optimize the matching result: Among them, w old represents the pre-parameter w c The parameter before update, w new represents the pre-parameter w c The parameter after update, is the gradient of the loss value with respect to the parameter, which determines the direction of parameter update, and η is the learning rate; the loss value loss(·) is the mutual information MI(A, B) between the filtered image W and the Laplacian pyramid residual image of the optical image corresponding to the Laplacian pyramid residual images, where A and B are the Laplacian pyramid residual images corresponding to the filtered image W and the optical image respectively corresponding to the Laplacian pyramid residual images; When the mutual information MI(A,B) is less than the mutual information threshold Γ, continue the iteration until the mutual information threshold Γ is reached; 3.

5. Use the diffeomorphic method for non-rigid matching.

2. The method for matching heterogeneous images of spatial targets based on image completion according to claim 1, wherein the mutual information MI(A, B) between the filtered image W and the Laplacian pyramid residual image corresponding to the optical image is as follows ​ MI(A,B)=H(A)+H(B)-H(AB) (8) Where Among them, A and B are the Laplacian pyramid residual images corresponding to the filtered image W and the optical image respectively. a and b are the corresponding pixel points of the images respectively; h(a, b) is the joint gray-level histogram between the images.

3. The method for matching heterogeneous images of space targets based on image completion according to claim 2, characterized in that, The process of constructing the Laplacian pyramid residual image described in step 3.3 includes the following steps: First, construct the Gaussian pyramid: Among them, G l-1 (·) represents the pyramid image of the (l - 1)-th layer; l represents the layer of the image, q and e are two variables; w(q, e) is a Gaussian template; Then perform the reverse operation. Through the interpolation operation, expand the image at a certain level in the Gaussian pyramid to the size of the previous level, which is expressed as follows: The image of this layer is obtained by subtracting the original image of the previous layer from the result of interpolating the image of the previous layer, that is, the residual image, G l+1 (i,j) is the representation of the (l + 1)-th layer, which subtracts the image of the previous layer to obtain the Laplacian pyramid residual image.

4. The method for matching heterogeneous images of space targets based on image completion according to claim 3, wherein, Before filtering the completed ISAR image and before converting the optical image into a grayscale image, it is necessary to filter the ISAR image and scale the optical image according to the markers in the image so that the spatial scale difference between the heterogeneous images does not exceed 10%.

5. The method for matching heterogeneous images of space targets based on image completion according to claim 4, wherein The process of filtering the completed ISAR image is realized by enhanced Lee filtering, Where, I represents the image pixels within the sliding window, is the mean value of the pixels within the sliding window, W is the estimation of the image, i.e., the filtered image; C I represents the local standard deviation coefficient of the image, C min = C u , L represents the equivalent number of looks, u represents the noise, C u is the local standard deviation coefficient of the noise; p is the weighting coefficient of the Lee filter.

6. The method for matching heterogeneous images of space targets based on image completion according to claim 5, wherein The process of constructing the ISAR image sequence including the spatial target described in step 2.1 includes the following steps: Select N consecutive ISAR image sequences, intercept a square area of size n×n at the center of the image, ensure that this area can completely contain the spatial target, and after interception, form the input data tensor X with a size of n×n×3×N, where 3 represents the three color channels of the ISAR image.

7. The method for matching heterogeneous images of space targets based on image completion according to claim 6, wherein The process of simulating the ISAR image sequence under the condition of target spin based on the three-dimensional spatial target model and the range-Doppler algorithm includes the following steps: 1.

1. First, open the 3D model file of the spatial target, and save it as a parasolid format file and an mdl file that are convenient for importing into FEKO software; 1.

2. Open the.x_t file in FEKO software, set the solution algorithm, target material, mesh quality, and electromagnetic wave carrier frequency parameters, establish the three-dimensional mesh of the target, and then save it as a.nas framework file; 1.

3. After saving the framework file as a text file, process the strings in the text file, export the grid point coordinates of the scatter point group to MATLAB, uniformly select N point coordinates from the output results, and make the target completely displayed in the ISAR image, so as to obtain the discrete scatter point distribution of the model; 1.

4. Use the range-Doppler algorithm to perform ISAR imaging on the scattering points with known distributions, and simulate the ISAR image sequence in the case of target spin.

8. The method for matching heterogeneous images of spatial targets based on image completion according to any one of claims 1 to 7, characterized in that, The construction process of the missing model includes the following steps: Assume that the pixel size of the ISAR image X is m×m×3, and some of its elements are missing. The observed index set is (i, j, k) ∈ Ω, where i, j, and k are the indices of the ISAR image pixels on the three channels respectively, and Ω is the observation set. Those not in this set are the missing pixels. Let the missing model S satisfy s ij = 1, (i, j, k) ∈ Ω, otherwise s ij = 0, Then the ISAR image after missing pixels can be expressed as

Images