SAR (Synthetic Aperture Radar) image fusion method, device and equipment in terahertz frequency band bunching mode

Abstract

The invention relates to an SAR image fusion method, device and equipment in a terahertz frequency band bunching mode, and the method comprises the steps: carrying out the preprocessing of a terahertz bunching mode video, and enabling a reference image and a to-be-fused image to be projected to a ground plane through combining the linearity and interpolation of a radar imaging digital elevation model, thereby obtaining a corresponding projection image; constructing an anisotropic diffusion scale space image for the two types of projection images, generating a Harris function scale space image according to the anisotropic diffusion scale space image, extracting extreme points, constructing SIFT feature descriptors of the two types of images based on the extreme points, obtaining an initial feature matching point pair set through similarity comparison, obtaining an optimized point pair set after multiple times of screening, and obtaining an anisotropic diffusion image through the optimized point pair set. And calculating an image conversion affine matrix by least square matching, fusing the to-be-fused image and the reference image according to the matrix to obtain a fused image, updating the image as the reference image, and repeatedly fusing the reference image with the next to-be-fused image until all frames are fused to obtain a final fused image. By adopting the method, the quality of the fused image can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of radar signal processing, in particular to a SAR image fusion method, device and equipment in a terahertz frequency band beamforming mode. BACKGROUND

[0002] In radar imaging, in order to suppress the coherent speckle noise of a SAR image and improve the signal-to-noise ratio of the image, a plurality of SAR images of the same scene obtained by the same imaging radar are combined into a clear SAR image through non-coherent superposition or image fusion. In order to obtain a non-coherent superposition image or a fusion image with high quality, the position accuracy of the image feature point pair is crucial. A slight deviation may make the quality of the fusion image worse.

[0003] At present, there are many methods for SAR image stitching, registration and fusion based on feature points. The main method is based on scale-invariant feature transform (SIFT), such as using a synthetic aperture radar-scale invariant feature transform (SAR-SIFT) algorithm, a position scale orientation-scale invariant feature transform algorithm (PSO-SIFT) and other improved SIFT algorithms. These methods are good for the stitching of optical images or strip mode SAR images, but for terahertz frequency band beamforming mode SAR images with serious geometric deformation, registration, stitching, fusion and image superposition are difficult problems. The main reason is that the imaging range of the terahertz frequency band video SAR image is small, the imaging quality of the terahertz video SAR is greatly affected by the radar platform motion error and the coherent speckle, especially the angle between the radar and the center of the imaging scene changes during the imaging process, and the resulting image has serious geometric deformation. Therefore, in the process of image stitching, registration and fusion of the terahertz frequency band beamforming mode SAR images obtained by the same radar for the same imaging scene, in addition to the misalignment of the positions of the feature point pairs, there are also image scaling and shearing. In the registration method based on feature points, random sample consensus (RANSAC) and fast sample consensus (FSC) are used for affine matrix parameter estimation of image conversion. This algorithm is an uncertain algorithm, and the obtained parameters are affected by the number of iterations, and the obtained results may not be the optimal solution. SUMMARY

[0004] Therefore, it is necessary to provide a SAR image fusion method, device and equipment in a terahertz frequency band beamforming mode, which can effectively improve the quality of the fusion image.

[0005] A SAR image fusion method in a terahertz frequency band beamforming mode, the method comprising: The SAR video in a terahertz frequency band beam mode is acquired, each frame SAR image in the SAR video is preprocessed, one frame SAR image is selected as a reference image, and other SAR images are all taken as to-be-fused images; A digital elevation model based on radar imaging is used to project the reference image and the to-be-fused images to a ground plane by a linear interpolation method to obtain a reference projection image and a to-be-fused projection image. After the reference projection image and the to-be-fused projection image are respectively subjected to Gaussian smoothing processing, an anisotropic diffusion scale space image corresponding to each is constructed, and a Harris scale space is constructed based on the anisotropic diffusion scale space image to generate a Harris function scale space image, and extreme points are extracted in the Harris function scale space image. SIFT feature descriptors of the reference projection image and the to-be-fused projection image are constructed based on the extreme points, an initial feature matching point pair set of the reference projection image and the to-be-fused projection image is obtained by similarity comparison of the SIFT feature descriptors, and the initial feature matching point pair is screened multiple times to obtain an optimized feature matching point pair set. Based on the optimized feature matching point pair set, an image conversion affine matrix is calculated by least square matching, the to-be-fused image and the reference image are fused according to the image conversion affine matrix to obtain a fused image, and the fused image is taken as the reference image to fuse with a next frame to-be-fused image until fusion of all to-be-fused images is completed, and a final SAR fused image is obtained.

[0006] In one embodiment, the SAR video includes two or more frames of SAR images.

[0007] In one embodiment, when the anisotropic diffusion scale space image corresponding to the reference projection image and the to-be-fused projection image is constructed, the following steps are included: The projection image filtered by the Gaussian window function is taken as a first layer image of the anisotropic diffusion scale space. After the first layer image is subjected to Gaussian filtering, a smoothing image is obtained, a first-order gradient of the smoothing image in x and y directions is calculated, a contrast factor is obtained according to the first-order gradient, a conduction coefficient is obtained by solving a diffusion equation, a second layer image is obtained by an additive operator splitting (AOS) algorithm, the next layer image is obtained based on the second layer image, and each layer image of the anisotropic diffusion scale space image is obtained by continuous iterative calculation.

[0008] In one embodiment, the diffusion equation is expressed as: ; In the above formula, represents a conduction function, represents a grayscale image, represents a gradient of represents a scale parameter, wherein the conduction function represents: In the above formula, represents a Gaussian smoothing image, represents a gradient of wherein the function is represented as: In the above formula, represents a contrast factor.

[0009] In one embodiment, the initial feature matching point pair set is screened multiple times to obtain an optimized feature matching point pair set, including: By comparing the positions, scales, layers and angles of the initial feature matching point pairs, the repeated feature matching point pairs in the initial feature matching point pair set are deleted to complete the first screening; In the initial feature matching point pair set after the first screening, three feature point pairs are randomly selected, an image affine matrix corresponding to the feature point pairs is constructed by least square matching, and based on the image affine matrix, the inliers of the matching feature point pairs are screened and the outliers are removed to complete the second screening; In the initial feature matching point pair set after the second screening, according to a preset distance threshold, the Euclidean distance between the matching feature point pairs is calculated to perform the last screening to obtain the optimized feature matching point pair set.

[0010] In one embodiment, after obtaining the final SAR fusion image, the SAR fusion image is also subjected to Gaussian window smoothing.

[0011] The application also provides a SAR image fusion device in a terahertz frequency band beamforming mode, the device comprising: A SAR video acquisition module is configured to acquire a SAR video in a terahertz frequency band beamforming mode, and after pre-processing each frame of SAR image in the SAR video, one frame of SAR image is selected as a reference image, and other SAR images are all taken as to-be-fused images. An image projection module is configured to project the reference image and the to-be-fused images to a ground plane based on a digital elevation model of radar imaging by a linear interpolation method to obtain a reference projection image and to-be-fused projection images. ​​An extreme point extraction module is configured to construct an anisotropic diffusion scale space image for each of the reference projection image and the to-be-fused projection image after Gaussian smoothing, and construct a Harris scale space based on the anisotropic diffusion scale space image, thereby generating a Harris function scale space image, and extract extreme points in the Harris function scale space image; An optimized feature matching point pair set obtaining module is configured to construct SIFT feature descriptors for the reference projection image and the to-be-fused projection image based on the extreme points, obtain an initial feature matching point pair set of the reference projection image and the to-be-fused projection image by similarity comparison of the SIFT feature descriptors, and obtain an optimized feature matching point pair set by multiple screenings of the initial feature matching point pair. A final SAR fusion image obtaining module is configured to calculate an image conversion affine matrix by least square matching based on the optimized feature matching point pair set, fuse the to-be-fused image and the reference image according to the image conversion affine matrix to obtain a fused image, and fuse the fused image as the reference image with a next frame of to-be-fused image until fusion of all to-be-fused images is completed, thereby obtaining a final SAR fusion image.

[0012] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above-mentioned SAR image fusion method in the terahertz frequency band in a beam mode when executing the computer program.

[0013] The above-mentioned SAR image fusion method, device and equipment in the terahertz frequency band in a beam mode, by pre-processing the terahertz beam mode SAR video, selecting one frame as a reference image and the rest as to-be-fused images, combining radar imaging digital elevation model linear and interpolation to project the reference and to-be-fused images to the ground plane to obtain corresponding projection images, respectively Gaussian smoothing the two types of projection images, constructing anisotropic diffusion scale space images, then generating Harris function scale space images, extracting extreme points, constructing SIFT feature descriptors of the two types of images based on the extreme points, obtaining an initial feature matching point pair set through similarity comparison, obtaining an optimized point pair set after multiple screenings, obtaining an image conversion affine matrix by least square matching through the optimized point pair set, fusing the to-be-fused and reference images according to the matrix to obtain a fused image, updating the image as the reference image, repeating the fusion with the next frame of to-be-fused image, until all frames are fused to obtain a final SAR fusion image. The method can solve the problems of position misalignment, image scaling and shearing caused by geometric deformation of the image, and poor image quality caused by the influence of the coherent spot in the image fusion of the terahertz frequency band video SAR image, thereby improving the quality of the fused image. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 Flowchart of SAR image fusion method in terahertz frequency band spotlight mode in one embodiment; Figure 2 Flowchart of SAR image fusion method in terahertz frequency band spotlight mode in one embodiment; Figure 3 SAR image in spotlight mode in one experiment, wherein, Figure 3 (a) represents a reference image, Figure 3 (b) represents a to-be-fused image; Figure 4 Projection image of SAR image in one experiment, Figure 4 (a) represents a reference projection image, Figure 4 (b) to-be-fused projection image; Figure 5 Optical schematic diagram of gnomon of imaging scene in one experiment; Figure 6 Overlay schematic diagram of SAR-SIFT algorithm in one experiment; Figure 7 Schematic diagram of fused image obtained by using the method in one experiment; Figure 8 Structural block diagram of SAR image fusion device in terahertz frequency band spotlight mode in one embodiment; Figure 9 Internal structural diagram of computer device in one embodiment. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0016] When the terahertz frequency band spotlight mode video SAR image is fused, if the traditional image stitching, registration and fusion algorithms such as SAR-SIFT and PSO-SIFT are used, there will be obvious image quality problems. From the imaging principle, the images to be fused are all irradiated by the same terahertz video SAR to the same target, and are generated by an advanced imaging algorithm. In theory, there should only be two geometric transformation relationships of translation and plane rotation between the images. However, in the actual fusion process, the affine matrix calculated by the feature point pair registration contains additional image scaling and shearing transformation components, which directly leads to a significant decline in the quality of the fused image. Further experimental verification found that even after the random sample consensus (RANSAC) algorithm processing, the error matching situation still remains in the selected feature point pair, which shows that the accuracy of the traditional registration algorithm cannot meet the high-quality fusion requirements of the terahertz frequency band spotlight mode video SAR image.

[0017] Through further investigation of the underlying reasons, on the one hand, the terahertz video SAR image in the spotlight mode is affected by the coherent speckle noise, resulting in a decrease in feature extraction accuracy; on the other hand, the difference in the angle between the imaging line of sight and the target surface will cause the imaging position of the target to shift in different frames of images, and then cause image geometric deformation. These two factors together make it difficult for traditional algorithms to achieve accurate registration, which ultimately affects the fusion effect.

[0018] In view of the above technical problems, in the present application, as shown in Figure 1 A terahertz frequency band spotlight mode SAR image fusion method is provided, which specifically includes the following steps: Step S100, a SAR video in the terahertz frequency band spotlight mode is obtained, after pre-processing each frame of SAR image in the SAR video, one frame of SAR image is selected as a reference image, and the other SAR images are all taken as images to be fused.

[0019] Step S110, based on the digital elevation model of radar imaging, the reference image and the image to be fused are projected to the ground plane by the linear interpolation method, to obtain the reference projection image and the image to be fused projection image.

[0020] Step S120, after the reference projection image and the image to be fused projection image are respectively subjected to Gaussian smoothing processing, the corresponding anisotropic diffusion scale space image is constructed, and the Harris scale space is constructed based on the anisotropic diffusion scale space image, so as to generate the Harris function scale space image, and then the extreme value points are extracted in the Harris function scale space image.

[0021] Step S130, SIFT feature descriptors of the reference projection image and the to-be-fused projection image are respectively constructed based on the extreme points, an initial feature matching point pair set of the reference projection image and the to-be-fused projection image is obtained by similarity comparison of the SIFT feature descriptors, and the initial feature matching point pair is screened for multiple times to obtain an optimized feature matching point pair set.

[0022] Step S140, based on the optimized feature matching point pair set, an image conversion affine matrix is calculated by least square matching, the to-be-fused image and the reference image are fused according to the image conversion affine matrix to obtain a fused image, and the fused image is taken as the reference image to fuse with the next frame of to-be-fused image until the fusion of all to-be-fused images is completed, and then a final SAR fusion image is obtained.

[0023] In step S100, the SAR video includes two or more frames of SAR images.

[0024] In this embodiment, when the SAR image fusion is performed, the multiple to-be-fused images are fused in time sequence based on the reference image.

[0025] In step S110, based on a digital elevation model (DEM) of radar imaging, the model contains the elevation information of the target region surface, and for each pixel in the reference image and the to-be-fused image, the corresponding position of the pixel in the horizontal plane coordinate system is calculated by linear interpolation combined with the position of the pixel in the radar coordinate system and the elevation data corresponding to the DEM. In this way, the reference image and the to-be-fused image are respectively mapped to the horizontal plane by pixel conversion, and finally the reference projection image and the to-be-fused projection image are obtained.

[0026] In step S120, when constructing the anisotropic diffusion scale space image corresponding to the reference projection image and the to-be-fused projection image respectively, the following steps are included: the projection image filtered by the Gaussian window function is taken as the first layer image of the anisotropic diffusion scale space, the smoothed image obtained by Gaussian filtering of the first layer image is calculated, the first order gradient of the smoothed image in x and y directions is calculated, the contrast factor is obtained according to the first order gradient, the conduction coefficient is obtained by solving the diffusion equation, the second layer image is obtained by the AOS algorithm, the next layer image is obtained based on the second layer image, and each layer image of the anisotropic diffusion scale space image is obtained by continuous iteration calculation, so that the anisotropic diffusion scale space image corresponding to the reference projection image and the to-be-fused projection image respectively is obtained.

[0027] In this embodiment, the diffusion equation is expressed as: ; In the above formula, represents the conduction function, represents a gray scale image, represents a gradient of represents a scale parameter, wherein the conduction function represents: In the above formula, represents a Gaussian smoothing image, represents a gradient of represents a contrast factor, wherein the function represents:

[0028] In the present embodiment, based on the obtained anisotropic diffusion scale space image, the Harris response function is calculated on different scale layers thereof, thereby constructing a multi-scale Harris scale space. Based on the scale space, a Harris function scale space image containing Harris response values of each scale is generated; finally, in the function image, the response values of each pixel and its neighborhood pixels are compared, and the pixels with the maximum or minimum response values are screened out, which are the extreme points to be extracted, and the principal directions of the extreme points are also calculated.

[0029] In step S130, the neighborhood pixels are first taken as the center of the extreme points of the reference projection image and the projection image to be fused, and the gradient information is calculated after alignment according to the principal direction, thereby generating the SIFT feature descriptor. Then, the similarity, for example, the Euclidean distance, between each SIFT descriptor of the reference projection image and all SIFT descriptors of the projection image to be fused is calculated, the descriptor pair with the highest similarity is screened out, and the initial feature matching point pair is formed correspondingly, and finally the initial feature matching point pair set of the two types of images is obtained. Then, the feature matching point pairs in the initial feature matching point pair set are screened.

[0030] In the present embodiment, the initial feature matching point pair set is screened multiple times to obtain the optimized feature matching point pair set, including: by comparing the position, scale, layer number and angle of the initial feature matching point pairs, the repeated feature matching point pairs in the initial feature matching point pair set are deleted, and the first screening is completed. In the initial feature matching point pair set after the first screening, three feature point pairs are randomly selected, the corresponding image affine matrix is constructed through least square matching, the inliers of the matching feature point pairs are screened based on the image affine matrix, and the outliers are removed, and the second screening is completed. In the initial feature matching point pair set after the second screening, the last screening is performed according to the preset distance threshold by calculating the Euclidean distance between the matching feature point pairs, thereby obtaining the optimized feature matching point pair set.

[0031] ​​​Specifically, in the first screening process, the position, scale, layer, and angle of the initial feature matching point pairs are used as the criteria for judgment, and the point pair set is checked one by one. If there are two or more point pairs, the position coordinates, scale layer, and angle parameters of the corresponding points in the reference projection image and the projection image to be fused are completely consistent (i.e., duplicate records), then one valid point pair is retained, and the remaining duplicates are deleted, completing the initial cleaning of redundant point pairs.

[0032] Specifically, in the second screening process, three sets of non-collinear feature point pairs are randomly selected from the point pair set after the first screening. An image affine matrix is ​​constructed using least squares matching (LSM). The reference image coordinates of all remaining point pairs are substituted into this matrix to calculate the theoretical coordinates of the image to be fused. The deviation between the theoretical coordinates and the actual matching coordinates is then compared. Points with deviations within a set range are considered interior points (i.e., points conforming to the geometric transformation rules), while those with deviations exceeding the range are considered exterior points (i.e., incorrectly matched points). Only interior points are retained for the next round. After repeating the second screening process multiple times, the second round of erroneous point pair removal is completed.

[0033] Specifically, in the third screening process, for the set of interior points after the second screening, a preset distance threshold is first set, and the Euclidean distance between each pair of matching feature points is calculated. If the distance is less than or equal to the preset threshold, it indicates that the two point pairs have high similarity and a reliable match, and they are retained. If the distance exceeds the threshold, they are judged as low-reliability matching point pairs and are directly deleted. The final set of retained point pairs is the optimized feature matching point pair set.

[0034] In step S140, after obtaining the final SAR fused image, it is also smoothed using a Gaussian window. A 3×3 Gaussian window smoothing process is applied to the final fusion result to eliminate edge noise and artifacts, resulting in the final SAR fused image.

[0035] like Figure 2 The diagram shown is a complete flowchart of this method.

[0036] This paper also demonstrates the effectiveness of the proposed method through experimental results. In the experiments, the spotlight SAR images were obtained from terahertz video SAR imaging, and the parameters of the imaging scene are shown in Table 1. The experimental procedure is as follows: Figure 2 As shown.

[0037] Table 1. Parameters of the measured data imaging

[0038] Figure 3Fig. 2 is a selected two terahertz spotlight mode SAR images. The two SAR images are the same terahertz video SAR images, the images are ground circular sundials, the radar has a certain angle with the ground, after imaging, an elliptical image is obtained, and there is a coherent spot in the image, the Arabic characters inside the ellipse are not clear, and the angle between the target and the radar imaging line of sight is different in different images.

[0039] Figure 4 Fig. 3 is a projection image of the image projected onto the ground plane. The projection image restores the circular shape of the ground and adds the frequency point labels of the coordinate axes. The projection image still has a serious influence of coherent spot, which makes the image unclear.

[0040] Figure 5 Fig. 4 is an optical image of the imaging scene sundial. Figure 5 There are sundials, steps around the sundials, Roman numeral characters, and green plants.

[0041] Figure 6 Fig. 5 is a fused image using the SAR-SIFT algorithm, and the image shows the overlapping traces of image fusion. The first frame image is taken as the reference frame, and the second frame image is taken as the frame to be fused. After the projection image of the second frame is fused with the projection image of the first frame reference frame, the obtained affine matrix is: ; From the parameters of the affine matrix, there is image scaling and image clipping between the images, which indicates that the target image has geometric deformation or mis-matching of feature point pairs. After false sample consistency (FSC) is removed, there are 19 feature point pairs. The image entropy of the fused image is: 12.474536.

[0042] Figure 7 Fig. 6 is a projection image fused by the method. The first frame image is taken as the reference frame, and the second frame image is taken as the frame to be fused. After the projection image of the first frame to be fused is fused with the reference projection image, the obtained affine matrix is: ; Similarly, the fused image processed by the method does not appear obvious image scaling, the rotation angle is 6.75, and the translation amount is After RANSAC removes the error point pairs, there are 16 matched feature point pairs, the image entropy is: 12.450894, and the structural similarity value is: 0.47543.

[0043] Compared with Figure 6 The method proposed in the present application can clearly see the Roman characters around the sundial and the rows of green trees on the sundial step edge. Figure 5 The image entropy of Figure 6 is 12.474536, which indicates that Figure 7the image quality of the image is better than Figure 6 the imaging quality of the image.

[0044] In the SAR image fusion method in the terahertz frequency band focusing mode, the images to be fused are projected to the ground plane by interpolation to obtain projected images; one of the images is selected as a reference projected image, and the others are to-be-fused projected images; Gaussian smoothing and anisotropy processing are respectively performed on the reference projected image and the to-be-fused projected images; after a Harris scale space is constructed, extreme points are extracted, and a descriptor of the position, histogram, and main direction of the extreme points is constructed; feature point pairs are screened through the Euclidean distance between the descriptors of the feature points; inner points that meet the affine matrix are screened through a least square matching (LSM) algorithm, and outer points that do not meet the condition are deleted; the positions of the screened feature point pairs are extracted, and the Euclidean distance of the positions of the feature point pairs is calculated to eliminate incorrect feature point pairs; the remaining feature point pairs are subjected to the LSM algorithm to obtain an affine matrix for image conversion; image fusion is performed through the affine matrix for image conversion to obtain a SAR image with high fusion quality; and the fused image is subjected to 3*3 Gaussian window smoothing to obtain an output image. According to the SAR imaging scene in the focusing mode, the image is projected to the ground imaging plane to correct the geometric deformation of the SAR image; the Euclidean distance of the positions of the feature point pairs is calculated to eliminate incorrect feature point pairs, so that a correct affine matrix is obtained, and the quality of the fused image is improved. Meanwhile, the anisotropic scale space is constructed, the extreme points of the SAR image can be extracted more accurately than in the Gaussian gradient scale space, and the descriptor of the position, histogram, and main direction of the feature point pairs is used to calculate the Euclidean distance of the feature points to screen the feature points, so that correct feature point pairs are not eliminated, and enough feature point pairs are reserved for subsequent matching.

[0045] It should be understood that, although Figure 1 the steps in the flowchart of the method are shown in the order of the arrows, the steps are not necessarily executed in the order of the arrows. Unless otherwise specified herein, the steps are not strictly limited in the order of execution, and the steps can be executed in other orders. Moreover, Figure 1 at least part of the steps in the method can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0046] In one embodiment, as Figure 8As shown, a SAR image fusion device under a terahertz frequency range beam mode is provided, comprising: a SAR video acquisition module 200, an image projection module 210, an extreme point extraction module 220, an optimized feature matching point pair set obtaining module 230, and a final SAR fusion image obtaining module 240, wherein: The SAR video acquisition module 200 is configured to acquire a SAR video under a terahertz frequency range beam mode, and after pre-processing each frame of SAR image in the SAR video, select one frame of SAR image as a reference image, and select other SAR images as to-be-fused images. The image projection module 210 is configured to project the reference image and the to-be-fused images to a ground plane based on a digital elevation model of radar imaging by a linear interpolation method to obtain a reference projection image and a to-be-fused projection image. The extreme point extraction module 220 is configured to perform Gaussian smoothing on the reference projection image and the to-be-fused projection image respectively, construct corresponding anisotropic diffusion scale space images, construct a Harris scale space based on the anisotropic diffusion scale space images, generate a Harris function scale space image, and extract extreme points in the Harris function scale space image. The optimized feature matching point pair set obtaining module 230 is configured to construct SIFT feature descriptors of the reference projection image and the to-be-fused projection image based on the extreme points, obtain an initial feature matching point pair set of the reference projection image and the to-be-fused projection image by similarity comparison of the SIFT feature descriptors, and perform multiple screenings on the initial feature matching point pair set to obtain an optimized feature matching point pair set. The final SAR fusion image obtaining module 240 is configured to calculate an image conversion affine matrix based on the optimized feature matching point pair set by least square matching, fuse the to-be-fused image and the reference image according to the image conversion affine matrix to obtain a fused image, and fuse the fused image as the reference image with a next frame of to-be-fused image until the fusion of all to-be-fused images is completed, and then obtain a final SAR fusion image.

[0047] The specific limitations of the SAR image fusion device under the terahertz frequency range beam mode can be referred to the limitations of the SAR image fusion method under the terahertz frequency range beam mode in the foregoing, which will not be repeated here. Each module in the above SAR image fusion device under the terahertz frequency range beam mode can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0048] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a SAR image fusion method in a terahertz band spotting mode. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0049] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0050] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Acquire SAR video in terahertz band spotting mode, preprocess each frame of SAR image in the SAR video, select one frame of SAR image as reference image, and then use the other SAR images as images to be fused. A digital elevation model based on radar imaging projects a reference image and an image to be fused onto the ground plane using linear interpolation, resulting in a reference projected image and an image to be fused. After Gaussian smoothing is applied to the reference projection image and the projection image to be fused respectively, an anisotropic diffusion scale space image is constructed. Based on the anisotropic diffusion scale space image, a Harris scale space is constructed to generate a Harris function scale space image. Then, extreme points are extracted from the Harris function scale space image. SIFT feature descriptors of the reference projection image and the to-be-fused projection image are respectively constructed based on the extreme points, an initial feature matching point pair set of the reference projection image and the to-be-fused projection image is obtained by similarity comparison of the SIFT feature descriptors, and the initial feature matching point pairs are screened multiple times to obtain an optimized feature matching point pair set; Based on the optimized feature matching point pair set, an image conversion affine matrix is calculated by least square matching, the to-be-fused image and the reference image are fused according to the image conversion affine matrix to obtain a fused image, and the fused image is taken as the reference image to fuse with a next frame of to-be-fused image until fusion of all to-be-fused images is completed, and then a final SAR fused image is obtained.

[0051] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium. The computer program is executed by a processor to implement the following steps: A SAR video in a terahertz frequency band beam mode is acquired, and after pre-processing of each frame of SAR image in the SAR video, one frame of SAR image is selected as a reference image, and other SAR images are taken as to-be-fused images; Based on a digital elevation model of radar imaging, the reference image and the to-be-fused image are projected to a ground plane by a linear interpolation method to obtain a reference projection image and a to-be-fused projection image; After the reference projection image and the to-be-fused projection image are respectively subjected to Gaussian smoothing processing, corresponding anisotropic diffusion scale space images are constructed, and a Harris scale space is constructed based on the anisotropic diffusion scale space images to generate a Harris function scale space image, and extreme points are extracted in the Harris function scale space image; SIFT feature descriptors of the reference projection image and the to-be-fused projection image are respectively constructed based on the extreme points, an initial feature matching point pair set of the reference projection image and the to-be-fused projection image is obtained by similarity comparison of the SIFT feature descriptors, and the initial feature matching point pairs are screened multiple times to obtain an optimized feature matching point pair set; Based on the optimized feature matching point pair set, an image conversion affine matrix is calculated by least square matching, the to-be-fused image and the reference image are fused according to the image conversion affine matrix to obtain a fused image, and the fused image is taken as the reference image to fuse with a next frame of to-be-fused image until fusion of all to-be-fused images is completed, and then a final SAR fused image is obtained.

[0052] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0053] The technical features of the above embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0054] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent protection of the present application should be subject to the appended claims.

Claims

1. A SAR image fusion method in terahertz band spotting mode, characterized in that, The method includes: Acquire SAR video in terahertz band spotting mode, preprocess each frame of SAR image in the SAR video, select one frame of SAR image as reference image, and then use the other SAR images as images to be fused. A digital elevation model based on radar imaging projects a reference image and an image to be fused onto the ground plane using linear interpolation, resulting in a reference projected image and an image to be fused. After Gaussian smoothing is applied to the reference projection image and the projection image to be fused respectively, an anisotropic diffusion scale space image is constructed. Based on the anisotropic diffusion scale space image, a Harris scale space is constructed to generate a Harris function scale space image. Then, extreme points are extracted from the Harris function scale space image. SIFT feature descriptors for the reference projection image and the projection image to be fused are constructed based on the extreme points. By comparing the similarity of the SIFT feature descriptors, an initial set of feature matching point pairs for the reference projection image and the projection image to be fused is obtained. The initial set of feature matching point pairs is then filtered multiple times to obtain an optimized set of feature matching point pairs. Based on the optimized feature matching point pair set, the image transformation affine matrix is ​​calculated by least squares matching. The image to be fused and the reference image are fused according to the image transformation affine matrix to obtain the fused image. The fused image is then used as the reference image and fused with the next frame image to be fused until the fusion of all images to be fused is completed, thus obtaining the final SAR fused image.

2. The SAR image fusion method in terahertz band spotting mode according to claim 1, characterized in that, The SAR video includes two or more SAR images.

3. The SAR image fusion method in terahertz band spotting mode according to claim 1, characterized in that, Constructing anisotropic diffusion-scale spatial images corresponding to the reference projection image and the projection image to be fused, respectively, includes: The projected image filtered by the Gaussian window function is used as the first layer image in the anisotropic diffusion scale space. After applying Gaussian filtering to the first layer image, a smooth image is obtained. The first-order gradients in the x and y directions of the smooth image are calculated, and the contrast factor is obtained based on the first-order gradients. The conduction coefficient is then obtained by solving the diffusion equation. The second layer image is obtained by splitting the image using the additive operator AOS algorithm. The next layer image is then obtained based on the second layer image. Through continuous iterative calculation, each layer image of the anisotropic diffusion scale space image is obtained.

4. The SAR image fusion method in terahertz band spotting mode according to claim 3, characterized in that, The diffusion equation is expressed as: ； In the above formula, Represents the transmission function. Represents a grayscale image. express gradient, The scale parameter is represented by the transfer function: ； In the above formula, Represents a Gaussian smoothed image. express The gradient of, where, The function is represented as: ； In the above formula, This represents the contrast factor.

5. The SAR image fusion method in terahertz band spotting mode according to any one of claims 1 to 4, characterized in that, The initial set of feature matching point pairs is filtered multiple times to obtain an optimized set of feature matching point pairs, including: By comparing the position, scale, number of layers, and angle of the initial feature matching point pairs, duplicate feature matching point pairs in the initial feature matching point pair set are deleted, thus completing the first screening. In the initial set of feature matching point pairs after the first screening, three feature point pairs are randomly selected, and the corresponding image affine matrix is ​​constructed by least squares matching. Based on the image affine matrix, the interior points of the matching feature point pairs are screened, and the exterior points are removed to complete the second screening. In the initial set of feature matching point pairs after the second screening, a final screening is performed by calculating the Euclidean distance between the matching feature point pairs according to a preset distance threshold, to obtain the optimized set of feature matching point pairs.

6. The SAR image fusion method in terahertz band spotting mode according to claim 5, characterized in that, After obtaining the final SAR fused image, it is also smoothed using a Gaussian window.

7. A SAR image fusion device in terahertz band spotting mode, characterized in that, The device includes: The SAR video acquisition module is used to acquire SAR video in terahertz band spotting mode. After preprocessing each frame of SAR image in the SAR video, one frame of SAR image is selected as the reference image, and the other SAR images are used as images to be fused. The image projection module is used for radar imaging-based digital elevation models. It projects the reference image and the image to be fused onto the ground plane using linear interpolation to obtain the reference projected image and the projected image to be fused. The extreme point extraction module is used to perform Gaussian smoothing on the reference projection image and the projection image to be fused respectively, construct the corresponding anisotropic diffusion scale space image, and construct the Harris scale space based on the anisotropic diffusion scale space image to generate the Harris function scale space image, and then extract extreme points in the Harris function scale space image. The module for optimizing the feature matching point pair set is used to construct SIFT feature descriptors for the reference projection image and the projection image to be fused based on the extreme points. By comparing the similarity of the SIFT feature descriptors, an initial feature matching point pair set for the reference projection image and the projection image to be fused is obtained. The initial feature matching point pairs are then filtered multiple times to obtain an optimized feature matching point pair set. The final SAR fusion image acquisition module is used to calculate the image transformation affine matrix based on the optimized feature matching point pair set through least squares matching, fuse the image to be fused and the reference image according to the image transformation affine matrix to obtain the fused image, and use the fused image as the reference image to fuse with the next frame of the image to be fused, until the fusion of all the images to be fused is completed, and then the final SAR fusion image is obtained.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images