A method and device for fusion of optical and ISAR images of space targets

Abstract

The present invention provides a method and device for fusing optical and ISAR images of space targets, relating to the technical field of image processing. The method comprises: registering an ISAR image and an optical image based on a high-resolution network and homography transformation to obtain a registered optical image, thereby making the registration of the ISAR image and the optical image more accurate; fusing a first low-frequency image and a second low-frequency image based on preset rules to obtain a fused low-frequency subband image; and fusing a first bandpass direction subband image and a second bandpass direction subband image to obtain a bandpass subband fused image. The low-frequency images can be fused based on preset rules including constructing an activity level measurement index using local energy, multi-scale morphological gradients, and fractal dimensions. The preset rules used for the fusion of the low-frequency images are more comprehensive, thereby providing more comprehensive feature information of the low-frequency images, more comprehensive details of the final fused image, and better visual effects.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and device for fusing optical and inverse synthetic aperture radar (ISAR) images of a space target. Background Art

[0002] Inverse synthetic aperture radar (ISAR) can continuously observe space targets around the clock, in all weather conditions, and at long distances, producing high-resolution, focused images of space targets, known as ISAR images. ISAR has become an effective method for imaging and monitoring space targets. ISAR images reflect the electromagnetic scattering characteristics of the target in the microwave frequency band, typically clearly revealing details such as edges, protrusions, and corners. With the increasing number of spacecraft launched into space, their types and structures are becoming increasingly complex, making it increasingly difficult to identify and track space targets solely based on ISAR images. Optical sensors, another primary method for spacecraft imaging and monitoring, reflect the electromagnetic scattering characteristics of the target in the visible light band. They typically provide a clear representation of the target's outline and morphological features, making them easier for the human eye to interpret and understand. Therefore, multi-sensor image fusion technology can effectively integrate the feature information from the ISAR and optical images of the target, enabling the effective utilization of both ISAR and optical images of the space target. This technology is of great significance for target identification and tracking, on-orbit state estimation, and three-dimensional structure reconstruction of space targets.

[0003] Currently, the fusion of ISAR and optical images of space targets typically employs scale-invariant feature transform matching algorithms and multimodal image fusion methods based on multiscale decomposition. However, due to differences in imaging principles, optical and ISAR images exhibit significant differences in grayscale values ​​and morphological features. This makes scale-invariant feature transform matching algorithms less applicable, resulting in inaccurate registration of ISAR and optical images. Multimodal image fusion based on multiscale decomposition often relies on single criteria, such as local energy or spatial frequency. This makes it difficult to accurately assess the activity levels of brightness information, target contours, and small amounts of texture detail contained in both ISAR and optical images, leading to a loss of detail in the fused image. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a method and apparatus for fusing optical and ISAR images of space targets, so as to solve the problems of inaccurate registration of ISAR images and optical images and missing details in the fused images.

[0005] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of the present invention provides a method for fusing optical and ISAR images of a space target, the method comprising:

[0007] Acquire ISAR images and optical images;

[0008] Based on the high-resolution network and homography transformation, the ISAR image and the optical image are registered to obtain the registered optical image;

[0009] Decomposing the ISAR image and the registered optical image according to a non-subsampled contourlet transform to obtain corresponding low-frequency images and bandpass directional sub-band images, wherein the low-frequency image includes a first low-frequency image corresponding to the ISAR image and a second low-frequency image corresponding to the registered optical image, and the bandpass directional sub-band images include a first bandpass directional sub-band image corresponding to the ISAR image and a second bandpass directional sub-band image corresponding to the registered optical image;

[0010] According to a preset rule, the first low-frequency image and the second low-frequency image are fused to obtain a fused low-frequency sub-band image, and the first bandpass direction sub-band image and the second bandpass direction sub-band image are fused to obtain a bandpass sub-band fused image, wherein the preset rule includes a rule for constructing an activity level measurement index through local energy, multi-scale morphological gradient, and fractal dimension;

[0011] The fused low-frequency sub-band image and the bandpass sub-band fused image are subjected to inverse non-subsampled contourlet transform to obtain the final fused image.

[0012] A second aspect of the present application provides a device for fusing optical and ISAR images of space targets, the device comprising:

[0013] Acquisition module, used to acquire ISAR images and optical images;

[0014] A registration module is used to register the ISAR image and the optical image based on the high-resolution network and homography transformation to obtain a registered optical image;

[0015] a decomposition module, configured to decompose the ISAR image and the registered optical image respectively according to a non-subsampled contourlet transform to obtain corresponding low-frequency images and bandpass directional sub-band images, wherein the low-frequency images include a first low-frequency image corresponding to the ISAR image and a second low-frequency image corresponding to the registered optical image, and the bandpass directional sub-band images include a first bandpass directional sub-band image corresponding to the ISAR image and a second bandpass directional sub-band image corresponding to the registered optical image;

[0016] a fusion module, configured to fuse the first low-frequency image and the second low-frequency image according to preset rules to obtain a fused low-frequency sub-band image, and to fuse the first bandpass direction sub-band image and the second bandpass direction sub-band image to obtain a bandpass sub-band fused image, wherein the preset rules include a rule for constructing an activity level measurement index through local energy, multi-scale morphological gradient, and fractal dimension;

[0017] The transformation module is used to perform inverse non-subsampled contourlet transformation on the fused low-frequency sub-band image and the bandpass sub-band fused image to obtain a final fused image.

[0018] A third aspect of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the optical and ISAR image fusion method of a space target according to the first aspect or any optional embodiment of the first aspect.

[0019] Compared with the existing technology, the optical and ISAR image fusion method and device of space targets provided by the present invention can use high-resolution networks and homography transformation to realize the registration of ISAR images and optical images, making the registration of ISAR images and optical images more robust, thereby making the registration of ISAR images and optical images more accurate; low-frequency images can be fused according to preset rules including constructing activity level measurement indicators through local energy, multi-scale morphological gradients and fractal dimensions. The preset rules used for the fusion of low-frequency images are more comprehensive, making the feature information considered for the low-frequency images more comprehensive, thereby making the details of the final fused image more comprehensive and the visual effect better. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0021] Figure 1 Schematic diagram showing the process of optical and ISAR image fusion method for space targets Figure 1 ;

[0022] Figure 2 Schematic diagram showing the process of optical and ISAR image fusion method for space targets Figure 2 ;

[0023] Figure 3 Schematically showing a schematic diagram of a space target being a model of Tiangong-1;

[0024] Figure 4A schematic diagram schematically shows the key points marked on the Tiangong-1 model;

[0025] Figure 5 Schematic diagrams of ISAR images, optical images, and registered optical images at three imaging moments are shown;

[0026] Figure 6 The figure schematically shows the comparison result of the final fused image at the first imaging moment;

[0027] Figure 7 The figure schematically shows the final fusion image comparison result at the second imaging moment;

[0028] Figure 8 The figure schematically shows the comparison result of the final fused image at the third imaging moment;

[0029] Figure 9 The structure of the optical and ISAR image fusion device of space targets is schematically shown. DETAILED DESCRIPTION

[0030] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0031] It should be noted that, unless otherwise specified, the technical or scientific terms used in the present invention should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0032] The method in the embodiment of the present invention is described in detail below.

[0033] Figure 1 The flowchart of the optical and ISAR image fusion method of a space target in an embodiment of the present invention is schematically shown. Figure 1 As shown, the method may include:

[0034] S101. Acquire ISAR images and optical images.

[0035] There are space targets in both ISAR images and optical images.

[0036] ISAR images reflect the electromagnetic scattering characteristics of observed space targets in the microwave band, and can usually clearly show details such as edges, protrusions, and corners on the target. Optical images reflect the electromagnetic scattering characteristics of observed space targets in the visible light band, and can usually intuitively show the outline and morphological characteristics of the observed target.

[0037] S102. Register the ISAR image and the optical image according to the high-resolution network and the homography transformation to obtain a registered optical image.

[0038] Specifically, the ISAR image and the optical image are registered according to the High Resolution Net (HRNet) and homography transformation obtained in step S101 to obtain a registered optical image.

[0039] Due to the different imaging principles, optical images and ISAR images have large differences in grayscale values ​​and morphological features. Therefore, it is necessary to align the ISAR images and optical images.

[0040] S103 , decomposing the ISAR image and the registered optical image respectively according to non-subsampled contourlet transform to obtain corresponding low-frequency images and bandpass direction sub-band images.

[0041] The low-frequency image includes a first low-frequency image corresponding to the ISAR image and a second low-frequency image corresponding to the registered optical image, and the bandpass directional sub-band image includes a first bandpass directional sub-band image corresponding to the ISAR image and a second bandpass directional sub-band image corresponding to the registered optical image.

[0042] Specifically, in order to more effectively fuse the scattering characteristics of the observed space target in the ISAR image and the optical image, the ISAR image is decomposed according to the non-subsampled contourlet transform to obtain the first low-frequency image L corresponding to the ISAR image. R and the first bandpass direction subband image According to the non-subsampled contourlet transform, the registered optical image is decomposed to obtain the second low-frequency image L corresponding to the registered optical image. O and the second bandpass direction subband image Where d is the orientation and l is the decomposition layer.

[0043] The non-subsampled contourlet transform consists of a non-subsampled Laplacian pyramid filter bank and a non-subsampled directional filter bank.

[0044] S104 , according to a preset rule, fuse the first low-frequency image and the second low-frequency image to obtain a fused low-frequency sub-band image, and fuse the first bandpass direction sub-band image and the second bandpass direction sub-band image to obtain a bandpass sub-band fused image.

[0045] Among them, the preset rules include rules that construct activity level measurement indicators through local energy, multi-scale morphological gradient and fractal dimension.

[0046] To analyze the information features contained in the first and second low-frequency images, a preset rule is set up to construct an activity level measurement index using local energy, multi-scale morphological gradient, and fractal dimension. The activity level measurement index can measure the information features in the first and second low-frequency images.

[0047] S105 , performing an inverse non-subsampled contourlet transform on the fused low-frequency sub-band image and the bandpass sub-band fused image to obtain a final fused image.

[0048] Specifically, an inverse non-subsampled contourlet transform is performed on the fused low-frequency sub-band image and the bandpass sub-band fused image obtained in step S104 to obtain a final fused image.

[0049] Based on the above Figure 1 As can be seen from the implementation method, the embodiment of the present invention obtains an ISAR image and an optical image; aligns the ISAR image and the optical image according to a high-resolution network and a homography transformation to obtain a registered optical image; decomposes the ISAR image and the registered optical image respectively according to a non-subsampled contourlet transform to obtain corresponding low-frequency images and bandpass directional subband images; according to a preset rule, the first low-frequency image and the second low-frequency image are fused to obtain a fused low-frequency subband image, and the first bandpass directional subband image and the second bandpass directional subband image are fused to obtain a bandpass subband fused image, wherein the preset rule is a rule including constructing an activity level measurement index through local energy, multi-scale morphological gradient and fractal dimension; and performs an inverse non-subsampled contourlet transform on the fused low-frequency subband image and the bandpass subband fused image to obtain a final fused image. In this way, the high-resolution network and homography transformation can be used to realize the registration of ISAR images and optical images, making the registration of ISAR images and optical images more robust, and thus making the registration of ISAR images and optical images more accurate; the low-frequency image and the bandpass subband image can be fused according to the preset rules including the construction of activity level measurement indicators through local energy, multi-scale morphological gradient and fractal dimension. The preset rules used for the fusion of low-frequency images are more comprehensive, making the feature information considered for the low-frequency image more comprehensive, and thus making the details of the final fused image more comprehensive and the visual effect better.

[0050] As a refinement and expansion of the above embodiment, Figure 2 The process of the optical and ISAR image fusion method of the space target in the embodiment of the present invention is as follows Figure 2 , see Figure 2As shown, an embodiment of the present invention provides a method for fusing optical and ISAR images of a space target, which may include:

[0051] S201. Acquire ISAR images and optical images.

[0052] Before acquiring ISAR images and optical images, it also includes:

[0053] Step A1: Based on the range-Doppler algorithm, perform two-dimensional high-resolution sequential imaging of the space target echo to obtain an ISAR image.

[0054] Specifically, the space target is continuously observed by inverse synthetic aperture radar to obtain the space target echo, and the space target echo is imaged in two dimensions with high resolution sequence according to the range-Doppler algorithm to obtain the ISAR image.

[0055] Step A2: Perform optical imaging on the space target according to a ray tracing algorithm to obtain an optical image.

[0056] Specifically, the space target is continuously observed by an optical sensor to obtain the space target, and the space target is optically imaged according to a ray tracing algorithm to obtain an optical image.

[0057] The following steps S202 to S204 are specific operations of registering the ISAR image and the optical image based on the high-resolution network and homography transformation to obtain a registered optical image.

[0058] S202: input the pixel coordinates of the key points marked by the training samples and the spatial targets in the training samples into the high-resolution network, train the high-resolution network, and obtain a trained high-resolution network.

[0059] The training samples include preset ISAR images and preset optical images.

[0060] The space target includes the main body of the space target and the sailboard of the space target. Figure 3 The schematic diagram shows that the space target is a model of Tiangong-1. Figure 3 The space target is the Tiangong-1 model, which includes the main body of the Tiangong-1 model and the sailboard of the Tiangong-1 model.

[0061] The key points marked on the space target include the key points marked on the main body of the space target and the key points marked on the sailboard of the space target. Figure 4 A schematic diagram of the key points marked on the Tiangong-1 model is shown schematically. The black dots are the marked key points. The space target is the Tiangong-1 model. There are key points marked on the main body of the Tiangong-1 model and key points marked on the sailboard of the Tiangong-1 model. Figure 4The key points of the three-dimensional Tiangong-1 model are mapped to the two-dimensional image to obtain the pixel coordinate information of the key points on the two-dimensional image, and the pixel coordinate information of the key points is input into the high-resolution network as a label.

[0062] The training samples including the preset ISAR image and the preset optical image, the key points marked by the main body of the space target and the key points marked by the sailboard of the space target corresponding to the preset ISAR image, and the key points marked by the main body of the space target and the key points marked by the sailboard of the space target corresponding to the preset optical image are used as the input of the high-resolution network, and the key points marked by the main body of the space target and the key points marked by the sailboard of the space target are used as the real labels to train the high-resolution network.

[0063] HRNet achieves the goal of enhancing semantic information and precise location information by running branches with multiple resolutions in parallel and continuously interacting with different branches. It can also always maintain high-resolution representation, making the predicted key point location information more accurate.

[0064] S203: Input the ISAR image and the optical image into a trained high-resolution network to detect corresponding at least four first key points, at least four second key points, at least four third key points, and at least four fourth key points.

[0065] Among them, at least four first key points are key points corresponding to the main body of the space target in the ISAR image, at least four second key points are key points corresponding to the sailboard of the space target in the ISAR image, at least four third key points are key points corresponding to the main body of the space target in the optical image, and at least four fourth key points are key points corresponding to the sailboard of the space target in the optical image.

[0066] The ISAR image and the optical image are input into a trained high-resolution network, and the trained high-resolution network is used to perform key point detection on the ISAR image and the optical image to extract at least four first key points corresponding to the main body of the space target in the ISAR image, at least four second key points corresponding to the sailboard of the space target in the ISAR image, at least four third key points corresponding to the main body of the space target in the optical image, and at least four fourth key points corresponding to the sailboard of the space target in the optical image.

[0067] S204. Perform a corresponding homography transformation on the optical image according to a first key point matrix corresponding to the at least four first key points, a second key point matrix corresponding to the at least four second key points, a third key point matrix corresponding to the at least four third key points, and a fourth key point matrix corresponding to the at least four fourth key points to obtain a registered optical image.

[0068] According to the premise of using the homography matrix, the key point pairs used to solve the transformation matrix should be on the same plane. Therefore, the key points of the main body of the space target and the key points on the sail of the space target extracted by the trained high-resolution network are on the same plane. Assume that the key point matrix of the extracted main body of the space target in the ISAR image is the first key point matrix The key point matrix of the extracted space target's sailboard in the ISAR image, that is, the second key point matrix, is The key point matrix of the extracted space target body in the optical image, i.e. the third key point matrix, is The key points of the extracted sailboard of the space target in the optical image, namely the fourth key point matrix, are:

[0069] Specifically, performing a corresponding homography transformation on the optical image according to a first key point matrix corresponding to at least four first key points, a second key point matrix corresponding to at least four second key points, a third key point matrix corresponding to at least four third key points, and a fourth key point matrix corresponding to at least four fourth key points to obtain a registered optical image includes:

[0070] Step B1: According to the first key point matrix and the third keypoint matrix Construct the first transformation matrix H body .

[0071] Among them, the first transformation matrix is ​​the transformation matrix corresponding to the main body of the space target.

[0072] Use the homography transformation principle to construct the first transformation matrix corresponding to the main body of the space target

[0073]

[0074] Step B2: According to the second key point matrix and the fourth keypoint matrix Construct the second transformation matrix H panel .

[0075] The second transformation matrix is ​​the transformation matrix corresponding to the sailboard of the space target.

[0076] The second transformation matrix corresponding to the sailboard of the space target is constructed using the principle of homography transformation.

[0077] Step B3: performing corresponding homography transformation on the optical image according to the first transformation matrix and the second transformation matrix to obtain a registered optical image.

[0078] Specifically, performing a corresponding homography transformation on the optical image according to the first transformation matrix and the second transformation matrix to obtain a registered optical image includes:

[0079] Step B31: Solve the first transformation matrix and the second transformation matrix respectively according to the random sampling consistency algorithm to obtain the corresponding first solved transformation matrix and second solved transformation matrix.

[0080] The homography transformation matrix includes the first transformation matrix and the second transformation matrix. Since the calculation of the homography transformation matrix uses a homogeneous coordinate system, the 3×3 homography transformation matrix actually has only 8 degrees of freedom. Therefore, at least 4 pairs of registered points are required to solve the homography transformation matrix. When the trained high-resolution network can detect more than or equal to four key points on the main body and sailboard of the space target, the random sample consensus algorithm (RANSAC) can be used to calculate the homography transformation matrix H. body and H panel Solve and obtain the corresponding first solution transformation matrix and second solution transformation matrix.

[0081] Step B32: performing corresponding homography transformation on the first solved transformation matrix and the second solved transformation matrix to obtain a registered optical image.

[0082] Figure 5 This diagram schematically illustrates an ISAR image, an optical image, and a registered optical image at three imaging times. Imaging was performed at three times within the observable interval: imaging time 1, imaging time 2, and imaging time 3. The first column shows the ISAR image, the second column shows the optical image, and the third column shows the registered optical image. The first row shows the ISAR image, optical image, and registered optical image at imaging time 1; the second row shows the ISAR image, optical image, and registered optical image at imaging time 2; and the third row shows the ISAR image, optical image, and registered optical image at imaging time 3. The registered optical image is obtained by registering the ISAR and optical images using a high-resolution network and homography.

[0083] S205 , decomposing the ISAR image and the registered optical image respectively according to non-subsampled contourlet transform to obtain corresponding low-frequency images and bandpass direction sub-band images.

[0084] The low-frequency image includes a first low-frequency image corresponding to the ISAR image and a second low-frequency image corresponding to the registered optical image, and the bandpass directional sub-band image includes a first bandpass directional sub-band image corresponding to the ISAR image and a second bandpass directional sub-band image corresponding to the registered optical image.

[0085] Step S205 is the same as step S103 and will not be described again here.

[0086] Steps S206 to S208 are operations of fusing the first low-frequency image and the second low-frequency image according to a preset rule to obtain a fused low-frequency sub-band image, and fusing the first bandpass direction sub-band image and the second bandpass direction sub-band image to obtain a bandpass sub-band fused image.

[0087] S206. Construct an activity level measurement index through local energy, multi-scale morphological gradient and fractal dimension.

[0088] The low-frequency subband image mainly contains the energy information, average grayscale intensity and a small amount of texture information of the source image. Local energy (LE), multi-scale morphological gradient (MSMG) and fractal dimension (FD) are usually used to measure the three information features in the low-frequency image.

[0089] Combining local energy, multi-scale morphological gradient and fractal dimension, the activity level measurement index f is constructed. MEF , which fully reflects the characteristic information of the low-frequency sub-band image.

[0090] Specifically, activity level measurement indicators are constructed through local energy, multi-scale morphological gradient and fractal dimension, including:

[0091] The activity level measurement index is constructed by using local energy, multi-scale morphological gradient and fractal dimension using the following first formula:

[0092] f MEF (i,j)=f LE (i,j)·f MSMG (i,j)·f FD (i,j);

[0093] Among them, f MEF (i,j) is the activity level measurement index, f LE (i,j) is the local energy, f MSMG (i, j) is the multi-scale morphological gradient, f FD (i, j) is the fractal dimension, and (i, j) is the pixel point of the first low-frequency image and the second low-frequency image.

[0094] Among the activity level metrics, local energy can reflect the change relationship between the current pixel and its neighboring pixels on the one hand, and the brightness information of the image on the other hand. The definition of local energy is as follows:

[0095]

[0096] Among them, f LE (i, j) is the local energy at the pixel point (i, j) of the low-frequency image, I is the low-frequency image matrix, i is the low-frequency image row index, j is the low-frequency image column index, 2M+1 and 2N+1 are the sizes of the local windows of the low-frequency image rows and columns respectively, p is the offset position of the window row, and q is the offset position of the window column.

[0097] Multi-scale morphological gradient can effectively measure the clarity of an image and has strong prediction and recognition capabilities. The mathematical expression of multi-scale morphological gradient is as follows:

[0098]

[0099] Among them, f MSMG (i, j) is the multi-scale morphological gradient, Wm = 1 / (2×m+1) is the weight of the gradient at scale m, n is the number of scales, G m (i,j) is the gradient information, G m (i,j) is defined as follows:

[0100]

[0101] Where I is the low-frequency image matrix, is the dilation operator in morphology, is the erosion operator in morphology, SE radius is the basic structure element r, and n is the number of scales.

[0102] For a small amount of texture information in low-frequency images, the fractal dimension is used to measure it. The contrast in a window of size (2ω+1)×(2ω+1) is calculated, which is called the fractal dimension of the central pixel of the window. The calculation formula of fractal dimension is as follows:

[0103] f FD (i,j)=g max (i,j)-g min (i,j);

[0104] Among them, f FD (i,j) is the fractal dimension, g max (i, j) is the maximum pixel value of the low-frequency image in the window, g min (i, j) is the minimum pixel value of the low-frequency image in the window, ω is the size of the window, Represents a positive integer.

[0105] S207 : Fusing the first low-frequency image and the second low-frequency image according to the activity level measurement index to obtain a fused low-frequency sub-band image.

[0106] The activity level measurement index includes a first activity level measurement index corresponding to the first low-frequency image and a second activity level measurement index corresponding to the second low-frequency image, and the fused low-frequency sub-band image includes a low-frequency sub-band component of the ISAR image and a low-frequency sub-band component of the optical image.

[0107] Specifically, according to the activity level measurement index, the first low-frequency image and the second low-frequency image are fused to obtain a fused low-frequency sub-band image, including:

[0108] Metrics based on the first activity level and a second activity level measure The first low-frequency image and the second low-frequency image are fused using the following second formula to obtain a fused low-frequency sub-band image:

[0109]

[0110] in, is the fused low-frequency sub-band image, is the low-frequency subband component of the ISAR image, is the low-frequency subband component of the optical image, is the first activity level measurement indicator, is the second activity level measurement indicator, and (i, j) is the pixel point of the first low-frequency image and the second low-frequency image.

[0111] S208 , fusing the first bandpass direction subband image and the second bandpass direction subband image according to an adaptive dual-channel pulse coupled neural network, fractal dimension, and maximum inter-class variance algorithm to obtain a bandpass subband fused image.

[0112] The first and second bandpass directional subband images are used as input to an adaptive dual-channel pulse-coupled neural network. This neural network consists of three components: a receptive field, an information fusion pool, and a pulse generator. The number of neurons in the neural network is equal to the number of pixels in the input source image.

[0113] The adaptive dual-channel pulse coupled neural network includes the internal behavior of neurons. The internal behavior of neurons in the adaptive dual-channel pulse coupled neural network can be calculated through fractal dimension and maximum inter-class variance algorithm, so that the network parameters of the adaptive dual-channel pulse coupled neural network can be adaptively calculated according to the input.

[0114] The internal behavior of neurons in an adaptive dual-channel pulse coupled neural network is calculated using the fractal dimension and maximum inter-class variance algorithm, including: using logarithmic functions for modeling, and combining the standard deviation and fractal dimension of the bandpass subband image to achieve the parameter α of the internal behavior of neurons u Adaptive calculation; using the standard deviation of the bandpass subband image and the threshold of the maximum inter-class variance algorithm to realize the parameter α of the internal behavior of the neuron u Adaptive computing.

[0115] The global coupling and pulse synchronization characteristics of the adaptive dual-channel pulse coupled neural network are conducive to the extraction and fusion of texture features of the bandpass sub-band images corresponding to the ISAR image and the optical image. Therefore, the fusion rule of the bandpass sub-band images is constructed using the following formula to obtain the bandpass sub-band fusion image:

[0116]

[0117] Among them, B F (i, j) is the bandpass sub-band fusion image, is the bandpass sub-band fusion image of the ISAR image, is the bandpass sub-band fusion image of the optical image, is the internal behavior of neurons in the adaptive dual-channel pulse-coupled neural network corresponding to the ISAR image, is the internal behavior of neurons in the adaptive dual-channel pulse-coupled neural network corresponding to the optical image, d is the orientation, l is the decomposition layer, and (i, j) is the pixel point of the first bandpass direction sub-band image and the second bandpass direction sub-band image.

[0118] S209 , performing an inverse non-subsampled contourlet transform on the fused low-frequency sub-band image and the bandpass sub-band fused image to obtain a final fused image.

[0119] The registration of ISAR images and optical images is achieved by using a key point monitoring high-resolution network and the principle of homography transformation, which is more robust. Different fusion rules are used for the different features contained in low-frequency images and bandpass sub-band images. The fusion rules designed for low-frequency image features consider feature information more comprehensively, resulting in better subjective visual effects of the final fused image and better objective evaluation indicators.

[0120] To verify the effectiveness of the algorithm proposed in this paper, this experiment conducted qualitative and quantitative analysis and compared it with several traditional algorithms, including Laplacian pyramid (LP), discrete wavelet transform (DWT), and dual-tree complex wavelet transform (DTCWT).

[0121] Figure 6 The figure schematically shows the comparison results of the final fused image at the first imaging moment, wherein (a) is an ISAR image, (b) is an optical image, (c) is a Laplace pyramid, (d) is a discrete wavelet transform, (e) is a dual complex wavelet transform, and (f) is the method of the present invention; the first imaging moment is imaging moment 1, wherein the red line represents the local magnified area of ​​the final fused image; Figure 7 The figure schematically shows the comparison results of the final fused image at the second imaging moment, wherein (a) is an ISAR image, (b) is an optical image, (c) is a Laplace pyramid, (d) is a discrete wavelet transform, (e) is a dual complex wavelet transform, and (f) is the method of the present invention; the second imaging moment is imaging moment 2, wherein the red line represents the local magnified area of ​​the final fused image; Figure 8 The figure schematically shows the comparison results of the final fused image at the third imaging moment, wherein Figure (a) is an ISAR image, Figure (b) is an optical image, Figure (c) is a Laplace pyramid, Figure (d) is a discrete wavelet transform, Figure (e) is a dual complex wavelet transform, and Figure (f) is the method of the present invention; the third imaging moment is imaging moment 3, wherein the red line represents the local magnified area of ​​the final fused image. Figure 6-Figure 8 These are the comparative experimental results of three imaging moments based on four fusion algorithms. The fusion results show that the proposed algorithm not only preserves the brightness information and contour features of the optical image, but also effectively integrates a large number of detailed features of the ISAR image, demonstrating superior visual quality. Other fusion algorithms (Laplacian pyramid, discrete wavelet transform, and dual-tree complex wavelet transform) all lose brightness information in the optical image, resulting in color distortion and artifacts, resulting in poor visual quality.

[0122] In order to conduct a comprehensive and objective performance comparison, this paper selects four evaluation indicators: Average Gradient (AG), Spatial Frequency (SF), Mutual Information (MI), and Visual Information Fidelity for Fusion (VIFF).

[0123] The index calculation results of the four fusion algorithms at three imaging moments are shown in the following table, which correspond to the simulation results in the qualitative analysis.

[0124] Table 1 Experimental index results at the first imaging moment

[0125]

[0126]

[0127] Table 2 Experimental index results at the second imaging moment

[0128] AG SF MI VIFF Laplace pyramid 0.9959 8.1821 0.4867 0.6963 Discrete Wavelet Transform 1.0036 8.0649 0.5266 0.5663 Dual complex wavelet transform 1.0162 7.9774 0.4656 0.5442 Method of the present invention 1.0548 8.3334 0.5432 0.8521

[0129] Table 3 Experimental index results at the third imaging moment

[0130] AG SF MI VIFF Laplace pyramid 0.6533 6.4757 0.4122 0.6934 Discrete Wavelet Transform 0.6626 6.3108 0.4546 0.5476 Dual complex wavelet transform 0.6788 6.2422 0.3889 0.5232 Method of the present invention 0.7126 6.6454 0.4975 0.8810

[0131] Across the three simulation results, the proposed method achieved the best performance across all four metrics. The mutual information evaluation metric demonstrates that the proposed method effectively preserves the complementary information of the source image. The image-based evaluation metrics of average gradient and spatial frequency demonstrate that the proposed method effectively fuses the texture information and contour features of the source image. The excellent performance of the human visual perception-based evaluation metric for fusion fidelity of visual information is consistent with the qualitative analysis results, demonstrating that the proposed method achieves superior visual effects.

[0132] Based on the same inventive concept, as an implementation of the above-mentioned method for fusing optical and ISAR images of a space target, an embodiment of the present invention further provides a device for fusing optical and ISAR images of a space target. Figure 9 is a structural diagram of the device in the embodiment of the present invention, see Figure 9 As shown, the device may include:

[0133] An acquisition module 901 is used to acquire ISAR images and optical images;

[0134] a registration module 902 for registering the ISAR image and the optical image acquired by the acquisition module 901 based on the high-resolution network and homography transformation to obtain a registered optical image;

[0135] a decomposition module 903 configured to decompose the ISAR image acquired by the acquisition module 901 and the optical image registered by the registration module 902, respectively, based on a non-subsampled contourlet transform, to obtain corresponding low-frequency images and bandpass directional sub-band images, wherein the low-frequency images include a first low-frequency image corresponding to the ISAR image and a second low-frequency image corresponding to the registered optical image; and the bandpass directional sub-band images include a first bandpass directional sub-band image corresponding to the ISAR image and a second bandpass directional sub-band image corresponding to the registered optical image.

[0136] a fusion module 904 configured to fuse the first low-frequency image and the second low-frequency image decomposed by the decomposition module 903 according to preset rules to obtain a fused low-frequency sub-band image, and to fuse the first bandpass direction sub-band image and the second bandpass direction sub-band image decomposed by the decomposition module 903 to obtain a bandpass sub-band fused image, wherein the preset rules include a rule for constructing an activity level measurement indicator using local energy, multi-scale morphological gradient, and fractal dimension;

[0137] The transformation module 905 is used to perform an inverse non-subsampled contourlet transform on the low-frequency sub-band image fused by the fusion module 904 and the bandpass sub-band fused image fused by the fusion module 904 to obtain a final fused image.

[0138] The registration module 902 is specifically configured to input pixel coordinates of key points marked by training samples and space targets in the training samples into a high-resolution network, train the high-resolution network, and obtain a trained high-resolution network, wherein the training samples include a preset ISAR image and a preset optical image; input the ISAR image and the optical image into the trained high-resolution network, detect corresponding at least four first key points, at least four second key points, at least four third key points, and at least four fourth key points, wherein the at least four first key points are key points corresponding to the main body of the space target in the ISAR image, the at least four second key points are key points corresponding to the sailboard of the space target in the ISAR image, the at least four third key points are key points corresponding to the main body of the space target in the optical image, and the at least four fourth key points are key points corresponding to the sailboard of the space target in the optical image; and perform a corresponding homography transformation on the optical image according to a first key point matrix corresponding to the at least four first key points, a second key point matrix corresponding to the at least four second key points, a third key point matrix corresponding to the at least four third key points, and a fourth key point matrix corresponding to the at least four fourth key points, to obtain a registered optical image.

[0139] The registration module 902 performs corresponding homography transformation on the optical image according to a first key point matrix corresponding to at least four first key points, a second key point matrix corresponding to at least four second key points, a third key point matrix corresponding to at least four third key points, and a fourth key point matrix corresponding to at least four fourth key points to obtain a registered optical image, including: constructing a first transformation matrix according to the first key point matrix and the third key point matrix, the first transformation matrix being the transformation matrix corresponding to the main body of the space target; constructing a second transformation matrix according to the second key point matrix and the fourth key point matrix, the second transformation matrix being the transformation matrix corresponding to the sailboard of the space target; and performing corresponding homography transformation on the optical image according to the first transformation matrix and the second transformation matrix to obtain a registered optical image.

[0140] The registration module 902 performs a corresponding homography transformation on the optical image according to the first transformation matrix and the second transformation matrix to obtain a registered optical image, including: solving the first transformation matrix and the second transformation matrix respectively according to the random sampling consistency algorithm to obtain the corresponding first solved transformation matrix and second solved transformation matrix; performing a corresponding homography transformation on the first solved transformation matrix and the second solved transformation matrix to obtain a registered optical image.

[0141] Fusion module 904 is specifically used to construct an activity level measurement index through local energy, multi-scale morphological gradient and fractal dimension; fuse the first low-frequency image and the second low-frequency image according to the activity level measurement index to obtain a fused low-frequency sub-band image; and fuse the first bandpass directional sub-band image and the second bandpass directional sub-band image according to an adaptive dual-channel pulse coupled neural network, fractal dimension and maximum inter-class variance algorithm to obtain a bandpass sub-band fused image.

[0142] The fusion module 904 constructs an activity level measurement index using the local energy, the multi-scale morphological gradient, and the fractal dimension, including: constructing the activity level measurement index using the local energy, the multi-scale morphological gradient, and the fractal dimension using the following first formula:

[0143] f MEF (i,j)=f LE (i,j)·f MSMG (i,j)·f FD (i,j);

[0144] Among them, f MEF (i,j) is the activity level measurement index, f LE (i,j) is the local energy, f MSMG (i, j) is the multi-scale morphological gradient, f FD (i, j) is the fractal dimension, and (i, j) is the pixel point of the first low-frequency image and the second low-frequency image.

[0145] The fusion module 904 fuses the first low-frequency image and the second low-frequency image according to the activity level measurement index to obtain a fused low-frequency sub-band image, including: fusing the first low-frequency image and the second low-frequency image according to the first activity level measurement index and the second activity level measurement index using the following second formula to obtain a fused low-frequency sub-band image:

[0146]

[0147] in, is the fused low-frequency sub-band image, is the low-frequency subband component of the ISAR image, is the low-frequency subband component of the optical image, is the first activity level measurement indicator, is the second activity level measurement index, (i, j) is the pixel point of the first low-frequency image and the second low-frequency image; the activity level measurement index includes the first activity level measurement index corresponding to the first low-frequency image and the second activity level measurement index corresponding to the second low-frequency image, and the fused low-frequency sub-band image includes the low-frequency sub-band component of the ISAR image and the low-frequency sub-band component of the optical image.

[0148] The device may also include: an imaging module, which is used to perform two-dimensional high-resolution sequence imaging of the space target echo according to the range-Doppler algorithm before acquiring the ISAR image and the optical image to obtain the ISAR image; and to perform optical imaging of the space target according to the ray tracing algorithm to obtain the optical image.

[0149] It should be noted that the description of the above embodiment of the apparatus for fusion of optical and ISAR images of space targets is similar to the description of the above-mentioned method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiment of the apparatus for fusion of optical and ISAR images of space targets according to the present invention, please refer to the description of the method embodiment.

[0150] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the method in one or more of the above embodiments.

[0151] It should be noted that the description of the computer-readable storage medium embodiment above is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the computer-readable storage medium embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0152] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for fusing optical and ISAR images of space targets, characterized in that: The optical and ISAR image fusion method of the space target includes: Acquire ISAR images and optical images; registering the ISAR image and the optical image according to a high-resolution network and a homography transformation to obtain a registered optical image; Decomposing the ISAR image and the registered optical image respectively according to a non-subsampled contourlet transform to obtain corresponding low-frequency images and bandpass directional sub-band images, wherein the low-frequency images include a first low-frequency image corresponding to the ISAR image and a second low-frequency image corresponding to the registered optical image, and the bandpass directional sub-band images include a first bandpass directional sub-band image corresponding to the ISAR image and a second bandpass directional sub-band image corresponding to the registered optical image; According to a preset rule, the first low-frequency image and the second low-frequency image are fused to obtain a fused low-frequency sub-band image, and the first bandpass directional sub-band image and the second bandpass directional sub-band image are fused to obtain a bandpass sub-band fused image, wherein the preset rule includes a rule for constructing an activity level measurement indicator by using local energy, multi-scale morphological gradient, and fractal dimension; performing an inverse non-subsampled contourlet transform on the fused low-frequency sub-band image and the bandpass sub-band fused image to obtain a final fused image; The registering the ISAR image and the optical image according to the high-resolution network and the homography transformation to obtain a registered optical image includes: Inputting pixel coordinates of key points marked by training samples and space targets in the training samples into a high-resolution network, training the high-resolution network, and obtaining a trained high-resolution network, wherein the training samples include a preset ISAR image and a preset optical image; Inputting the ISAR image and the optical image into the trained high-resolution network, detecting corresponding at least four first key points, at least four second key points, at least four third key points, and at least four fourth key points, wherein the at least four first key points are key points corresponding to the main body of the space target in the ISAR image, the at least four second key points are key points corresponding to the sailboard of the space target in the ISAR image, the at least four third key points are key points corresponding to the main body of the space target in the optical image, and the at least four fourth key points are key points corresponding to the sailboard of the space target in the optical image; According to the first key point matrix corresponding to the at least four first key points, the second key point matrix corresponding to the at least four second key points, the third key point matrix corresponding to the at least four third key points, and the fourth key point matrix corresponding to the at least four fourth key points, the optical image is subjected to a corresponding homography transformation to obtain the registered optical image.

2. The method for fusion of optical and ISAR images of space targets according to claim 1, characterized in that: The step of performing a corresponding homography transformation on the optical image according to a first key point matrix corresponding to the at least four first key points, a second key point matrix corresponding to the at least four second key points, a third key point matrix corresponding to the at least four third key points, and a fourth key point matrix corresponding to the at least four fourth key points to obtain the registered optical image includes: Constructing a first transformation matrix according to the first key point matrix and the third key point matrix, where the first transformation matrix is ​​a transformation matrix corresponding to the main body of the space object; constructing a second transformation matrix according to the second key point matrix and the fourth key point matrix, where the second transformation matrix is ​​a transformation matrix corresponding to the sailboard of the space target; According to the first transformation matrix and the second transformation matrix, a corresponding homography transformation is performed on the optical image to obtain the registered optical image.

3. The method for fusion of optical and ISAR images of space targets according to claim 2, characterized in that: The step of performing a corresponding homography transformation on the optical image according to the first transformation matrix and the second transformation matrix to obtain the registered optical image includes: Solving the first transformation matrix and the second transformation matrix respectively according to a random sampling consensus algorithm to obtain corresponding first solved transformation matrix and second solved transformation matrix; Perform corresponding homography transformation on the first solved transformation matrix and the second solved transformation matrix to obtain the registered optical image.

4. The method for fusion of optical and ISAR images of space targets according to claim 1, characterized in that: The step of fusing the first low-frequency image and the second low-frequency image according to a preset rule to obtain a fused low-frequency sub-band image, and fusing the first bandpass direction sub-band image and the second bandpass direction sub-band image to obtain a bandpass sub-band fused image, includes: constructing the activity level measurement index by using the local energy, the multi-scale morphological gradient and the fractal dimension; fusing the first low-frequency image and the second low-frequency image according to the activity level measurement indicator to obtain the fused low-frequency sub-band image; The first bandpass directional subband image and the second bandpass directional subband image are fused according to an adaptive dual-channel pulse coupled neural network, a fractal dimension and a maximum inter-class variance algorithm to obtain the bandpass subband fused image.

5. The method for fusion of optical and ISAR images of space targets according to claim 4, characterized in that: The constructing the activity level measurement index by using the local energy, the multi-scale morphological gradient and the fractal dimension includes: The activity level measurement index is constructed by using the local energy, the multi-scale morphological gradient and the fractal dimension using the following first formula: ； in, is the activity level measure, is the local energy, is the multi-scale morphological gradient, is the fractal dimension, are pixels of the first low-frequency image and the second low-frequency image.

6. The method for fusion of optical and ISAR images of space targets according to claim 4, characterized in that: The activity level measurement index includes a first activity level measurement index corresponding to the first low-frequency image and a second activity level measurement index corresponding to the second low-frequency image, and the fused low-frequency sub-band image includes a low-frequency sub-band component of the ISAR image and a low-frequency sub-band component of the optical image; The fusing the first low-frequency image and the second low-frequency image according to the activity level measurement indicator to obtain the fused low-frequency sub-band image includes: The first low-frequency image and the second low-frequency image are fused according to the first activity level measurement indicator and the second activity level measurement indicator using the following second formula to obtain the fused low-frequency sub-band image: ； in, is the fused low-frequency sub-band image, is the low-frequency sub-band component of the ISAR image, is the low-frequency sub-band component of the optical image, is the first activity level measurement indicator, is the second activity level measurement indicator, are pixels of the first low-frequency image and the second low-frequency image.

7. The method for fusion of optical and ISAR images of space targets according to claim 1, characterized in that: Before acquiring the ISAR image and the optical image, the optical and ISAR image fusion method of the space target further includes: Performing two-dimensional high-resolution sequential imaging of space target echoes according to a range-Doppler algorithm to obtain the ISAR image; The space target is optically imaged according to a ray tracing algorithm to obtain the optical image.

8. A device for fusion of optical and ISAR images of space targets, characterized in that: The optical and ISAR image fusion device of the space target includes: Acquisition module, used to acquire ISAR images and optical images; a registration module, configured to register the ISAR image and the optical image based on a high-resolution network and a homography transformation to obtain a registered optical image; a decomposition module, configured to decompose the ISAR image and the registered optical image respectively according to a non-subsampled contourlet transform to obtain corresponding low-frequency images and bandpass directional sub-band images, wherein the low-frequency images include a first low-frequency image corresponding to the ISAR image and a second low-frequency image corresponding to the registered optical image, and the bandpass directional sub-band images include a first bandpass directional sub-band image corresponding to the ISAR image and a second bandpass directional sub-band image corresponding to the registered optical image; a fusion module, configured to fuse the first low-frequency image and the second low-frequency image according to preset rules to obtain a fused low-frequency sub-band image, and to fuse the first bandpass direction sub-band image and the second bandpass direction sub-band image to obtain a bandpass sub-band fused image, wherein the preset rules include a rule for constructing an activity level measurement indicator using local energy, multi-scale morphological gradient, and fractal dimension; a transform module, configured to perform an inverse non-subsampled contourlet transform on the fused low-frequency sub-band image and the bandpass sub-band fused image to obtain a final fused image; The registration module is specifically used to input the pixel coordinates of the key points marked by the training samples and the space target in the training samples into the high-resolution network, train the high-resolution network, and obtain a trained high-resolution network, wherein the training samples include a preset ISAR image and a preset optical image; input the ISAR image and the optical image into the trained high-resolution network, and detect corresponding at least four first key points, at least four second key points, at least four third key points, and at least four fourth key points, wherein the at least four first key points are the key points corresponding to the main body of the space target in the ISAR image; The at least four second key points are the key points corresponding to the sailboard of the space target in the ISAR image, the at least four third key points are the key points corresponding to the main body of the space target in the optical image, and the at least four fourth key points are the key points corresponding to the sailboard of the space target in the optical image; according to the first key point matrix corresponding to the at least four first key points, the second key point matrix corresponding to the at least four second key points, the third key point matrix corresponding to the at least four third key points, and the fourth key point matrix corresponding to the at least four fourth key points, the optical image is subjected to corresponding homography transformation to obtain the registered optical image.

9. A computer-readable storage medium, characterized in that The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the optical and ISAR image fusion method of a space target according to any one of claims 1 to 7.

Images