Massive Multi-Band Image Generation Method for Machine Learning in Space-Based Target Detection

Abstract

This invention relates to a method for generating massive multi-band images using machine learning for space-based target detection. The method calculates the optical properties of a 3D target, fuses the target's optical image into a 2D background, and can quickly simulate multi-band optical images. Addressing the problem of insufficient training samples in multi-band machine learning target detection algorithms, this paper proposes a fast calculation method for target-background fusion. This method includes simulating the background's optical image in multiple bands, adding a target optical property image with a shadow effect to the background using a 3D model optical property calculation method, and simulating different time conditions through meteorological parameter settings and temperature calculations. Experiments show that this method can quickly simulate multi-band optical images fused with 3D targets while maintaining the accuracy of optical property calculation.

Description

Technical Field

[0001] This invention relates to image processing technology and radiometric calculation technology, specifically to a method for generating massive multi-band images using machine learning for space-based target detection. Background Technology

[0002] Remote sensing images play a vital role in both civilian and military fields. With the rapid development of deep neural networks, the application of remote sensing images in target detection, segmentation, tracking, and recognition has been greatly promoted. However, due to limitations in hardware capabilities and cost, remote sensing images in certain bands (such as the mid-infrared band) currently suffer from difficulties in acquisition and low resolution, and the demand for massive training samples in deep learning cannot be met.

[0003] Currently, the mainstream method for remote sensing image simulation involves using simulation software platforms to perform scene simulation. This involves analyzing and modeling the target and scene to obtain a 3D model, and then calculating the optical image of the band of interest based on optical radiation theory and the remote sensing imaging link model. Compared to using real-world field images, image simulation software not only reduces costs but also allows for simulations under arbitrary parameter conditions. However, this method requires modeling both the background and the target, with large-scale background modeling and calculation being particularly time-consuming and labor-intensive. Furthermore, deep learning-based target detection methods primarily focus on extracting target features, so high-precision optical feature simulation of the background often results in wasted time and computational costs, making it unsuitable for rapidly generating large batches of remote sensing images for machine learning. Summary of the Invention

[0004] To overcome the problems of complex scene modeling and large parameters in traditional remote sensing image simulation, which result in excessively high computation costs for a large number of fine background features in target detection methods, this invention proposes a massive multi-band image generation method for space-based target detection using machine learning.

[0005] This invention proposes a massive multi-band image generation method for space-based target detection using machine learning. This method rapidly calculates large-scale background optical characteristics images in the band of interest by utilizing image spectral information and a spectral reflectance database. Combined with a three-dimensional model radiation calculation method, it adds a three-dimensional target optical image with a shadow effect to the background. Furthermore, it achieves rapid simulation of images under different time conditions through meteorological parameter settings and scene temperature distribution calculations. The specific process is as follows:

[0006] Step 1: Input the remote sensing base map and determine the image type. Perform reflectance inversion and land cover classification on the multispectral base map. Complete the category determination and reflectance query of the band of interest based on the land cover reflectance database. Perform land cover classification and manually set the reflectance of the band of interest on the panchromatic base map.

[0007] Step 2: Based on the base map feature classification results in Step 1, establish a simplified model of typical scenarios including the target, and calculate the temperature distribution of the target and various types of features in thermal simulation software by combining the required climate conditions and time parameters.

[0008] Step 3: Input detector and ground coordinate data, combine with the reflectivity base map of the band of interest obtained in Step 1, obtain the background reflectivity image under the camera imaging range through the payload geometric imaging module, and calculate the background radiance image based on the meteorological conditions and temperature distribution results in Step 2.

[0009] Step 4: Input the 3D target model and ground model, combine the coordinate data in Step 3, the meteorological conditions and temperature distribution in Step 2, obtain the reflected solar radiation brightness and thermal radiation brightness of the model through the 3D model optical property calculation module, use MODTRAN to calculate the atmospheric radiation brightness and the model reflected environmental radiation brightness, and superimpose all radiation parts to obtain a 3D target radiation brightness image with shadow effect.

[0010] Step 5: Spatially fuse the background and target radiance images obtained in Steps 3 and 4, and use the modulation transfer function to add detector performance effects to the entrance pupil radiance image, finally obtaining the simulated image of the band of interest.

[0011] Furthermore, step 4 specifically involves:

[0012] First, import the target and ground 3D models. The ground model attributes are the parameters of the target placement location selected in the background image. Divide the model into facets and determine the normal vectors and position attributes of the facets. Based on the normal vector information of the model facets and the camera observation conditions, determine the illumination vector and camera observation vector of the model facets. Then, using the illumination and observation vectors from step 4.1, as well as the relative position information of the facets, determine the dual-visible facets that receive solar radiation and can be observed by the detector. Further, calculate the reflected solar radiation brightness and thermal radiation brightness emitted by each facet towards the detector. Finally, use MODTRAN to calculate the atmospheric radiation brightness and reflected environmental radiation brightness, and superimpose all radiation components to obtain a 3D target radiation brightness image with a shadow effect.

[0013] The present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0014] This invention is applicable to the problem of rapidly generating training samples of massive optical images in multi-band machine learning. Under the premise of ensuring the correct overall radiance distribution of the simulated image, it can quickly simulate optical images of fused three-dimensional targets for different time conditions and different observation conditions, and has good applicability. Attached Figure Description

[0015] Figure 1 This is the overall flowchart of the present invention;

[0016] Figure 2 This is a temperature field image of a typical airport scene under summer midday conditions;

[0017] Figure 3 This is a B-52 bomber model image to be integrated into the background image;

[0018] Figure 4 This is a graph showing the calculated optical (mid-infrared) characteristics of the target model;

[0019] Figure 5 This is a target shadow effect diagram in the visible light band;

[0020] Figure 6 It is a simulated mid-wave infrared image;

[0021] Figure 7 This is a comparison image of a real-world remote sensing image in the visible spectrum and a simulated image. Detailed Implementation

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0023] This invention proposes a massive multi-band image generation method for space-based target detection using machine learning. This method, by fusing two-dimensional background and three-dimensional target imaging, can rapidly complete optical image simulation of large-scale scenes. The process is as follows: Figure 1 As shown.

[0024] The specific process of the multi-band image simulation method for fusing three-dimensional targets and two-dimensional backgrounds is as follows:

[0025] Step 1: Input the remote sensing base map and determine the image type. Based on the land cover reflectance database, complete the category determination and query the reflectance of the band of interest. Perform land cover classification and manually set the reflectance of the band of interest for the panchromatic base map.

[0026] Step 2: Based on the land cover classification results of the remote sensing base map, establish a simplified model of typical scenes including the target, and calculate the meteorological conditions and temperature distribution of the target and various land cover types in thermal simulation software by combining the required climate conditions and time parameters;

[0027] Step 3: Input detector and ground coordinate data, combine with the reflectivity base map of the band of interest obtained in Step 1, obtain the background reflectivity image under the camera imaging range through the payload geometric imaging module, and calculate the background radiance image based on the meteorological conditions and temperature distribution results in Step 2.

[0028] Step 4: Input the 3D target model and ground model, combine the coordinate data in Step 3, the meteorological conditions and temperature distribution in Step 2, obtain the reflected solar radiation brightness and thermal radiation brightness of the model through the 3D model optical property calculation module, use MODTRAN to calculate the atmospheric radiation brightness and the model reflected environmental radiation brightness, and superimpose all radiation parts to obtain a 3D target radiation brightness image with shadow effect.

[0029] Step 5: Spatially fuse the background and three-dimensional target radiance images obtained in Steps 3 and 4, and use the modulation transfer function to add detector performance effects to the target and background radiance images at the detector entrance pupil, finally obtaining the simulated image of the band of interest.

[0030] Furthermore, step 4 specifically involves:

[0031] Step 4.1: Input the 3D target ground fusion model, which includes a target model and a ground model. The ground model attributes are the parameters of the target placement location selected in the background image.

[0032] The 3D target ground fusion model is divided into surface elements, and the normal vector and position attributes of the surface elements are determined. Based on the normal vector information of the model surface elements and the camera observation conditions, the lighting vector of the model surface elements and the camera observation vector are determined.

[0033] Step 4.2: Using the model element illumination vector and camera observation vector from Step 4.1, as well as the relative position information of the elements, determine the dual-visible elements that can both receive solar radiation and be observed by the detector; this prepares for calculating the reflected solar radiation brightness and thermal radiation brightness of the model through the 3D model optical property calculation module based on the material properties and meteorological parameters of the elements.

[0034] Step 4.3: Calculate the reflected solar radiation brightness and thermal radiation brightness emitted by each dual-visible element towards the detector;

[0035] Step 4.4: Calculate the atmospheric radiance and the reflected environmental radiance using MODTRAN, and then overlay all the radiative components to obtain a three-dimensional target radiance image with a shadow effect.

[0036] Furthermore, in step 2,

[0037] For temperature distribution calculation, a typical scene 3D model needs to be established, including all types of land features and targets on the remote sensing base map. An environmental model is added to obtain the typical scene and environmental model. The attributes of each part of the typical scene and environmental model are defined, including initializing the model space, inputting the material properties of each underlying surface component, and inputting the wind and solar attributes according to meteorological parameters. Then, the calculation area is divided into grids, the number of calculation iteration steps and residual convergence are set, and the calculation equation and discretization method are selected. Finally, the calculation is performed according to the radiation heat transfer equation and the results are post-processed to obtain the surface radiation temperature distribution map of the typical scene and environmental model. This map is used as a reference to set the target temperature and the temperatures of various land features in the base map.

[0038] The method of this application will be further introduced below with specific examples. The specific implementation process of the massive multi-band image generation method for space-based target detection machine learning disclosed in this application is as follows:

[0039] Step 1: After performing precise radiometric calibration and atmospheric correction (MODTRAN) on the multispectral base map, the reflectance images of ground objects in each known band can be obtained. Then, the reflectance images are segmented using the k-means clustering algorithm. The average reflectance of each type of ground object is compared with the ground object reflectance database to find the ground object material with the smallest overall error. The reflectance (emissivity) image in the required band is obtained by querying the spectral reflectance of the material. The panchromatic map is directly segmented using the k-means clustering algorithm. The classification results are manually identified and the reflectance (emissivity) image in the required band is obtained based on the ground object reflectance database.

[0040] Step 2: Establish a typical scene 3D model that includes all types of land features and targets on the remote sensing base map. Import the model into the PHOENICS software and add models such as the earth, wind, and sun. Define the attributes of each part of the model, including initializing the model space, inputting the material properties of each underlying surface component, and inputting the wind and sun attributes according to meteorological parameters. Then, divide the calculation area into grids, set the number of calculation iterations and residual convergence, select the calculation equation and discretization method, and finally perform the calculation and post-process the results to obtain the surface radiation temperature distribution map of the model. This map will be used as a reference to set the target temperature and the temperature of various land features in the base map.

[0041] Step 3: Input the detector coordinates and ground location coordinates, combine the reflectivity base map and camera parameters obtained in Step 1, obtain the corresponding projection relationship and transformation matrix under the geometric imaging model of the platform load, and calculate the ground coordinates corresponding to each pixel. Determine the imaging range of the camera in the large-scale base map, obtain the background reflectivity (emissivity) image, and combine the meteorological parameters and ground object temperature distribution calculation results in Step 2 to calculate the background radiance map of the camera through the radiation calculation module.

[0042] Step 4: Input the 3D target model and ground model and divide them into surface elements. Determine the normal vector and position of the surface elements. Next, based on the surface element normal vector information and camera observation conditions, determine the illumination and camera observation vectors of each surface element. Then, based on the illumination and observation vectors and the relative position information of the surface elements, determine the "dual visible" surface elements that can receive solar radiation and be observed by the detector. Based on the material properties and meteorological parameters of the surface elements, calculate the reflected solar radiation brightness and thermal radiation brightness of the model through the 3D model optical property calculation module. Use MODTRAN to calculate the atmospheric radiation brightness and the model reflected environmental radiation brightness. Superimpose all radiation parts to obtain a target radiation brightness image with shadow effect.

[0043] Step 5: Superimpose the background and target radiance images obtained in Steps 3 and 4 at selected locations, and then use the spatial modulation transfer function to add image shift, diffraction, and noise (Gaussian noise) effects to the entrance pupil radiance image, finally obtaining the simulation image of the desired band.

[0044] The following is an example of a multi-band image simulation method for fusing 3D targets with 2D backgrounds:

[0045] Input Worldview-3 multispectral image data and perform a simulation of the airport scene in the base image. Figure 3 This is a B52 bomber model, whose surface is divided into multiple triangular facets. Considering that this method is applied to the simulation of large-scale scenes, the accuracy of the target model can be simplified to a certain extent (the number of facets is adjusted to about 3000) in order to reduce the calculation time. Now, the method of this invention is applied to simulate the mid-infrared band (wavelength 3-5μm) images of the background and target under given parameters (meteorological conditions, camera parameters and coordinate data).

[0046] Application Step 1: Radiometric calibration and atmospheric correction are performed on the multispectral base map to convert the grayscale image into a reflectance image for the corresponding band. Then, k-means clustering is used to segment land features on the reflectance image. The average reflectance of each land feature in different bands is compared with the material library to find the one with the smallest overall error as the land feature material, and the emissivity of the material in the required band is queried. Here, the number of land feature classifications is set to 16. The relative root mean square error of the reflectance of each land feature obtained from the classification with the corresponding reflectance in the spectral reflectance database is shown in Table 1. Except for seawater, the relative error of each land feature is less than 1%, which is analyzed to be caused by the lack of seawater reflectance data in the standard material library, resulting in matching error. The overall result is within the usable range.

[0047] Table 1. Land cover classification matching error

[0048]

[0049]

[0050] Application Step Two: The simulation background image is a typical airport scene. A two-dimensional scene model is established, including typical objects such as grass, asphalt roads, cement roads, bare soil, cement-roofed buildings, steel-roofed buildings, and aluminum aircraft. The model is imported into the PHOENICS thermal simulation software. The Chen-Kimk-ε turbulence model and the IMMERSOL radiative heat equation based on the Stefan-Boltzmann law are selected for calculation. The number of calculation iterations is set to 1000. The temperature distribution of this scene under mid-latitude summer atmospheric data (in this experiment, the atmospheric type is set to mid-latitude summer atmosphere, the aerosol type is rural type VIS=23km, the solar zenith angle is 1°, and the solar azimuth angle is 170°) is simulated as follows: Figure 2 As shown;

[0051] Application Step 3: In the STK software, set the detector orbital altitude to 20km, the camera elevation angle α = 60°, the azimuth angle β = 90°, select the camera's staring center on the Earth's surface as the ground location coordinates, input the detector and ground location coordinates and reflectivity base map, calculate the imaging range of the base map in the camera, obtain the background reflectivity image, combine the meteorological parameters and ground object temperature distribution in Step 2 to set the temperature of various ground objects (18~32℃), and calculate the background radiance map of the camera using MODTRAN;

[0052] Application Step Four: Input the 3D target model, transform the coordinates of the target and detector to the target's body coordinate system, and then perform observation geometry calculations and visibility assessment. Set the target's spectral emissivity to ε = 0.9 and surface temperature to T = 30℃. Combined with meteorological parameters, use the 3D model optical property calculation module to obtain the target's reflected solar radiation brightness and thermal radiation brightness. The calculation results are as follows: Figure 4 As shown in (a), atmospheric radiance and target-reflected ambient radiance are calculated using MODTRAN, and all radiant components are superimposed to obtain the target's radiance image. The result is as follows. Figure 4 As shown in (b); due to the weak reflection effect of infrared light, the visible light band of 0.45-0.9 micrometers was selected for the shadow effect experiment. The reflectivity of the ground model is ref ground =0.08, Solar zenith angle α sun =45°, solar azimuth angle β sun = -45°, B-52 aircraft model reflectivity ref tar =0.15, the aircraft's orientation is assumed to be due north, and other meteorological parameters are the same as those in the infrared band simulation. The calculation results are as follows: Figure 5 As shown.

[0053] Application Step 5: Select a suitable location in the background radiance image obtained in Step 3, and overlay the target radiance image obtained in Step 4 onto the selected background location. Considering the influence of detector performance, image shift, diffraction, and noise effects are added to the target and background radiance images at the detector entrance pupil through modulation transfer function. Finally, a simulated image of the desired band is obtained, as shown in the figure. Figure 6 As shown.

[0054] Figure 7 To compare the real-world visible spectrum remote sensing images with the simulated images, since there is currently no publicly available high-resolution mid-wave and long-wave infrared remote sensing data, images of the Norfolk Naval Base airport in the United States, taken by the Worldview-3 satellite, were used. Seven bands (425nm–832nm) of data from the multispectral image were used to simulate the 950nm band image. The simulation image was verified to have an 8-bit quantization bit. The root mean square error (RMSE) was calculated to be 7.6323% by randomly selecting 100 corresponding pixels from the two images. This shows a good degree of conformity in terms of radiation range and ground feature detail. Considering that simulation conditions and shooting conditions cannot be completely identical, this RMS error is considered to be within a reasonable range and meets the simulation requirements.

[0055] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating massive multi-band images for space-based target detection machine learning, characterized in that, The method includes: Step 1: Input the remote sensing base map and determine the image type. Based on the land cover reflectance database, complete the category determination and query the reflectance of the band of interest. Perform land cover classification and manually set the reflectance of the band of interest for the panchromatic base map. Step 2: Based on the land cover classification results of the remote sensing base map, establish a simplified model of typical scenes including the target, and calculate the meteorological conditions and temperature distribution of the target and various land cover types in thermal simulation software by combining the required climate conditions and time parameters; Step 3: Input detector and ground coordinate data, combine with the reflectivity base map of the band of interest obtained in Step 1, obtain the background reflectivity image under the camera imaging range through the payload geometric imaging module, and calculate the background radiance image based on the meteorological conditions and temperature distribution results in Step 2. Step 4.1: Input the 3D target ground fusion model, which includes a target model and a ground model. The ground model attributes are the parameters of the target placement location selected in the background image. The 3D target ground fusion model is divided into surface elements, and the normal vector and position attributes of the surface elements are determined. Based on the normal vector information of the model surface elements and the camera observation conditions, the lighting vector of the model surface elements and the camera observation vector are determined. Step 4.2: Using the model element illumination vector and camera observation vector from Step 4.1, as well as the relative position information of the elements, determine the dual-visible elements that both receive solar radiation and can be observed by the detector. Step 4.3: Calculate the reflected solar radiation brightness and thermal radiation brightness emitted by each dual-visible element towards the detector; Step 4.4: Calculate the atmospheric radiance and the reflected environmental radiance using MODTRAN, and then superimpose all radiant components to obtain a three-dimensional target radiance image with a shadow effect; Step 5: Spatially fuse the background radiance image obtained in Step 3 with the three-dimensional target radiance images obtained in Steps 4.1 to 4.4, and use the modulation transfer function to add detector performance effects to the target and background radiance images at the detector entrance pupil, finally obtaining the simulated image of the band of interest.

2. The method of claim 1, wherein, In step 2, a typical scene 3D model containing all types of land features and targets in the remote sensing base map is established, and an environmental model is added to obtain the typical scene and environmental model; the attributes of each part of the typical scene and environmental model are defined, including initializing the model space, inputting the material properties of each underlying surface component, and inputting the wind and solar attributes according to meteorological parameters; then the computational area is divided into grids, the number of computation iteration steps and residual convergence are set, and the computational equation and discretization method are selected; Finally, calculations were performed based on the radiation heat transfer equation, and the results were post-processed to obtain a surface radiation temperature distribution map of a typical scene and environment model. This map was then used as a reference to set the target temperature and the temperature of various ground features in the base map.

Images