Ship target cross-view-angle image fusion method based on ISAR image and optical image

Abstract

The invention discloses a ship target cross-view-angle image fusion method based on an ISAR (inverse synthetic aperture radar) image and an optical image, and aims to solve the problems that ship target images of different view angles are not distinguished in an existing ship target recognition method, and only the unique advantages of a radar image and the optical image are combined into a new image for recognition; and whether the visual angles in the two image domains have deviation is not considered, so that the converted image has a distortion phenomenon due to lack of certain visual angle information, and the ship target recognition rate is low. The method comprises the following steps: acquiring an ISAR image and an optical image corresponding to a ship target according to a three-dimensional model of the ship target; and inputting the preprocessed ISAR image and the preprocessed optical image into a Selection GAN network, and outputting a fused image of the ship target. The preprocessed optical image is an optical image under an expected view angle. The visual angle information difference existing in the radar image and the optical image is used for guiding image conversion of the radar image and the optical image, feature fusion of the two image domains is achieved, and therefore the recognition rate of the ship target is increased. Belongs to the technical field of image fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a multi-modal cross-view image fusion method of a ship target based on ISAR images and optical images. Background Art

[0002] When observing ship targets, whether using radar or optical imaging, images of the ship target from different perspectives are acquired as the observation platform or the ship target moves. These images often contain more information about the ship target than a single image. Existing research on ship image conversion has not deliberately distinguished between images of ship targets from different perspectives. Instead, radar and optical images of ship targets are treated as images in two different domains and converted between them. The goal is to improve the recognition rate of ship targets by preserving and fusing the unique advantages of radar and optical images to form a new image. However, radar and optical images contain information from different perspectives. During the image conversion process, if the optical image domain contains perspectives of the ship target that are not present in the radar image domain, converting the radar image from the new perspective to the existing optical image perspective will result in the loss of some perspective information, causing distortion in the converted image, which can reduce the recognition rate of ship targets to a certain extent. Summary of the Invention

[0003] To address the problem that existing ship target recognition methods fail to distinguish ship target images from different perspectives, this invention simply combines the unique advantages of radar and optical images into a new image for recognition without considering any deviation in the perspectives of the two image domains. This causes distortion in the converted image due to the lack of certain perspective information, leading to low ship target recognition rates. Therefore, a cross-perspective image fusion method for ship targets based on ISAR and optical images is proposed.

[0004] The technical solution adopted by the present invention is:

[0005] It includes the following steps:

[0006] S1. Acquire several ship targets and generate a three-dimensional model of each ship target using three-dimensional modeling;

[0007] S2. Based on the three-dimensional model of each ship target, the ray tracing method is used to obtain the spatial coordinates and scattering intensity of each ship target in the ISAR imaging scene, and the radar parameters and motion parameters of each ship target are set;

[0008] Based on the obtained spatial coordinates and scattering intensity of the scattering points, as well as the set radar parameters and motion parameters, the range Doppler map algorithm is used to perform imaging simulation on each ship target, generate ISAR images of each ship target at different viewing angles, and obtain all ISAR images of all ship targets;

[0009] S3. Based on the three-dimensional model of each ship target, obtain the optical image of each ship target at different viewing angles to obtain all the optical images of all the ship targets;

[0010] S4, preprocessing all the ISAR images obtained in S2 and all the optical images obtained in S3 respectively, merging all the preprocessed ISAR images and all the optical images as a simulation sample library, and dividing the simulation sample library into a training set and a test set;

[0011] The pre-processed optical image includes an optical image at a conventional viewing angle and an optical image at a desired viewing angle;

[0012] S5. Input the training set into the SelectionGAN network for training, output the fused image of the ship target, and obtain the trained SelectionGAN network;

[0013] The test set is input into the trained SelectionGAN network and the fused image of the ship target is output.

[0014] Furthermore, the motion parameters of each ship target in S2 include the sea state level, and the ship position is set at the center of the scene.

[0015] Furthermore, the desired viewing angle in S4 is a custom viewing angle.

[0016] Furthermore, in S4, all the ISAR images obtained in S2 and all the optical images obtained in S3 are preprocessed respectively. The specific process is as follows:

[0017] For each ISAR image, denoising and normalization processing are performed in sequence to generate a preprocessed ISAR image, and all preprocessed ISAR images are obtained;

[0018] For each optical image, grayscale processing and normalization processing are performed in sequence to generate a preprocessed optical image, thereby obtaining all preprocessed optical images;

[0019] All preprocessed ISAR images and all preprocessed optical images are combined as a simulation sample library, and 80% of the samples in the simulation sample library are used as a training set and 20% of the samples are used as a test set.

[0020] Furthermore, the denoising process adopts an adaptive filtering algorithm.

[0021] Furthermore, the normalization process includes center normalization process and size normalization process.

[0022] Furthermore, when the training set is input into the SelectionGAN network for training in S5, the number of training samples in each batch is 32, a total of 7200 batches are trained, the learning rate is 0.00005, and the Adam optimization algorithm is used.

[0023] The beneficial effects of the present invention are:

[0024] The present invention first obtains radar images of the same ship target at different perspectives and optical images at the desired perspective, and then uses the perspective information difference between the radar image and the optical image to guide the image conversion process between the two. That is, the perspective information of the ship target that exists in the optical image domain but does not exist in the radar image domain is combined with the SelectionGAN network to guide the cross-perspective conversion process of the radar image. Finally, the feature fusion of the two image domains is realized, thereby improving the recognition rate of the ship target.

[0025] This paper proposes the first application of the SelectionGAN network to multimodal, cross-view image fusion of ship targets. By integrating multi-view ISAR and optical images with the SelectionGAN network, the problem of multimodal view information asymmetry in ship targets is effectively addressed. The SelectionGAN network is used to transform the ISAR image feature space using the perspective prior of the optical image, achieving feature complementarity between the two images to improve the quality of the fused image and generate an enhanced fused image. This solves the conversion distortion problem caused by the lack of view information in existing fused new images, thereby improving the recognition rate of ship targets. Simulations have verified that the recognition rate of ship targets has been increased to 50%, demonstrating its significant engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart for constructing a simulation sample library;

[0027] Figure 2 This is a schematic diagram of the SelectionGAN network working;

[0028] Figure 3 It is the image fusion flow chart of the present invention;

[0029] Figure 4 It is the overall flow chart of the present invention; DETAILED DESCRIPTION

[0030] Specific implementation method 1: Combination Figures 1-4This embodiment describes a method for cross-view image fusion of a ship target based on an ISAR image and an optical image, which includes the following steps:

[0031] S1. Acquire a plurality of ship targets and generate a three-dimensional model of each ship target using three-dimensional modeling. The plurality of ship targets is greater than or equal to two ship targets.

[0032] S2. Based on the three-dimensional model of each ship target, the ray tracing method is used to obtain the spatial coordinates and scattering intensity of each ship target in the ISAR imaging scene, and the radar parameters and motion parameters of each ship target are set. The motion parameters include the sea level and the position of the ship is set at the center of the scene. The present invention sets the sea level to level 2.

[0033] Based on the obtained spatial coordinates and scattering intensities of the scattering points, as well as the set radar and motion parameters, the range-Doppler map algorithm was used to simulate the imaging of each ship target. This generated ISAR images of each ship target at different viewing angles (original viewing angles), resulting in a complete set of ISAR images for all ship targets. The ISAR imaging results were nearly consistent with the measured images.

[0034] S3. Based on the three-dimensional model of each ship target, optical images of each ship target at different viewing angles are acquired to obtain all optical images of all ship targets.

[0035] S4. Preprocess all ISAR images obtained in S2 and all optical images obtained in S3, respectively. Combine all preprocessed ISAR images and all optical images as a simulation sample library, and divide the simulation sample library into a training set and a test set. The preprocessed optical images include optical images at a conventional viewing angle and optical images at a desired viewing angle, where the desired viewing angle is a custom viewing angle. The specific process is as follows:

[0036] For each ISAR image, denoising processing and normalization processing are performed in sequence to generate a preprocessed ISAR image, and all preprocessed ISAR images are obtained; the denoising processing adopts an adaptive filtering algorithm.

[0037] For each optical image, grayscale processing and normalization processing are sequentially performed to generate a preprocessed optical image, and all preprocessed optical images are obtained. The normalization processing includes center normalization processing and size normalization processing.

[0038] All preprocessed ISAR images and all preprocessed optical images are combined as a simulation sample library, and 80% of the samples in the simulation sample library are used as a training set, and the remaining 20% of the samples are used as a test set.

[0039] Combine Figure 1 As shown, the present invention simulates ship targets through 3D modeling. Based on the constructed 3D ship target model, ISAR images and optical images of the ship target at different viewing angles are acquired, enabling all-around, full-view analysis of the ship target and facilitating better observation of changes in the ship target at different viewing angles. Preprocessing is performed to convert the ISAR and optical images to the corresponding size format and to images at the desired viewing angle. This enriches the simulated sample library of the ship target, facilitates subsequent image fusion, and improves the recognition rate of the ship target.

[0040] S5. Input the training set into the SelectionGAN network for training, output the fused image of the ship target, and obtain the trained SelectionGAN network;

[0041] The test set is input into the trained SelectionGAN network and the fused image of the ship target is output.

[0042] SelectionGAN network (Multi-Channel Attention Selection GAN with CascadedSemantic GuidanceforCross-ViewImageTranslation) is a method based on multi-channel attention selection image translation under cascaded semantic guidance, which includes two generation processes, such as Figure 2 As shown in the first stage of the semantic graph-guided generative subnetwork, the input image I from the source perspective is first received. a (ISAR image of the present invention) and semantic graph S from the target perspective g (optical images of the present invention), and input them into the image generator G i , output the synthesized target perspective image I' g =G i (I a ,S g In this way, the reconstruction of the input semantic graph is further promoted, and the real semantic graph provides stronger supervision for guiding the cross-view conversion in the deep network, which promotes the theoretical optimization of the network. g is fed into the semantic generator G s The reconstructed semantic graph S' is obtained g , this process can be formalized as follows:

[0043] S' g =G s (I' g )=G s (G i (I a,S g ))

[0044] The optimization goal of the network is to make S' g As close to S as possible g , that is, a semantic generation loop is formed, which can be expressed as follows:

[0045] [I a ,S g ]→I' g →S' g ≈S g

[0046] G i and G s The two generators are explicitly connected by the real semantic graph, which provides additional constraints for the generator to ensure better learning of semantic structure consistency in the first stage. i Only coarse output can be produced, that is, only blurred target image details are produced and there is inconsistency with the target image at the pixel level. Based on this, the SelectionGAN network introduces a cascade model to explore the output from coarse to fine. In the second stage, it uses a new multi-channel attention selection module to better utilize the coarse output of the first stage and produce a fine-grained final output, significantly improving the output quality.

[0047] The multi-channel attention selection module consists of multi-scale spatial pooling and multi-channel attention selection components. First, to capture all the spatial information required for fine-grained generation, multi-scale spatial pooling is introduced to obtain multi-scale features. Based on the given coarse input and the coarse features derived from the first stage, the multi-scale spatial pooling is obtained and input into the multi-channel selection module. The multi-channel attention mechanism then automatically performs spatial and temporal selection from each generated image to synthesize a fine-grained final output.

[0048] During SelectionGAN network training, the number of training samples per batch is 32, for a total of 7,200 batches. The learning rate is 0.00005, and the Adam optimization algorithm is used. The SelectionGAN network training process follows the principle of alternating gradient descent between the discriminator and the generator. First, the discriminator is fixed while the generator is trained. Then, the generator is fixed and the discriminator is trained, resulting in an end-to-end optimization process. After training is complete, the semantic segmentation maps of the original view image (ISAR image) and the target view image (optical image) need only be fed into the network to obtain the converted target view image.

[0049] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A cross-view image fusion method for ship targets based on ISAR images and optical images, characterized by: It includes the following steps: S1. Acquire several ship targets and generate a three-dimensional model of each ship target using three-dimensional modeling; S2. Based on the three-dimensional model of each ship target, the ray tracing method is used to obtain the spatial coordinates and scattering intensity of each ship target in the ISAR imaging scene, and the radar parameters and motion parameters of each ship target are set; Based on the obtained spatial coordinates and scattering intensity of the scattering points, as well as the set radar parameters and motion parameters, the range Doppler map algorithm is used to perform imaging simulation on each ship target, generate ISAR images of each ship target at different viewing angles, and obtain all ISAR images of all ship targets; S3. Based on the three-dimensional model of each ship target, obtain the optical image of each ship target at different viewing angles to obtain all the optical images of all the ship targets; S4, preprocessing all the ISAR images obtained in S2 and all the optical images obtained in S3 respectively, merging all the preprocessed ISAR images and all the optical images as a simulation sample library, and dividing the simulation sample library into a training set and a test set; The pre-processed optical image includes an optical image at a conventional viewing angle and an optical image at a desired viewing angle; S5. Input the training set into the SelectionGAN network for training, output the fused image of the ship target, and obtain the trained SelectionGAN network; The test set is input into the trained SelectionGAN network and the fused image of the ship target is output.

2. The cross-view image fusion method for ship targets based on ISAR images and optical images according to claim 1, characterized in that: The motion parameters of each ship target in S2 include the sea state level and the position of the ship is set at the center of the scene.

3. The cross-view image fusion method for ship targets based on ISAR images and optical images according to claim 1, characterized in that: The desired viewing angle in S4 is a custom viewing angle.

4. The cross-view image fusion method for ship targets based on ISAR images and optical images according to claim 1, characterized in that: In S4, all the ISAR images obtained in S2 and all the optical images obtained in S3 are preprocessed respectively. The specific process is as follows: For each ISAR image, denoising and normalization processing are performed in sequence to generate a preprocessed ISAR image, and all preprocessed ISAR images are obtained; For each optical image, grayscale processing and normalization processing are performed in sequence to generate a preprocessed optical image, thereby obtaining all preprocessed optical images; All preprocessed ISAR images and all preprocessed optical images are combined as a simulation sample library, and 80% of the samples in the simulation sample library are used as a training set and 20% of the samples are used as a test set.

5. The method for cross-view image fusion of ship targets based on ISAR images and optical images according to claim 4, characterized in that: The denoising process adopts an adaptive filtering algorithm.

6. The method for cross-view image fusion of ship targets based on ISAR images and optical images according to claim 4, characterized in that: The normalization process includes center normalization process and size normalization process.

7. The cross-view image fusion method for ship targets based on ISAR images and optical images according to claim 1, characterized in that: When the training set is input into the SelectionGAN network for training in S5, the number of training samples in each batch is 32, a total of 7200 batches are trained, the learning rate is 0.00005, and the Adam optimization algorithm is used.

Images