A batch conversion method of Suomi NPP night light remote sensing data from H5 to GeoTiff format

Abstract

The present invention relates to the technical field of night light remote sensing data processing, and in particular to a method for batch conversion of Suomi NPP night light remote sensing data from H5 to GeoTiff format, comprising the following steps: step one: batch acquisition of H5 file names; step two: setting conversion parameters; step three: reading H5 file structure; step four: reading designated variable data; step five: geographic coordinate correction; step six: outputting GeoTiff data. The present invention can ensure that the quality of the converted data meets the standard and will not increase data noise; and can automatically convert all H5 files in a designated folder. Compared with the manual conversion mode using ENVI software, the present invention can significantly improve the conversion efficiency; it can not only convert the sensor data record (SDR) of Suomi NPP night light remote sensing data, but also convert the geometric data (Geo), and has stability. Compared with the use of ENVI software that can only perform SDR conversion, the present invention has significant advantages in geometric data conversion.

Description

Technical Field

[0001] The present invention relates to the technical field of night light remote sensing data processing, and in particular to a method for batch conversion of Suomi NPP night light remote sensing data in H5 format to GeoTiff format. Background Art

[0002] Suomi NPP (National Polar-orbiting Partnership) satellite night light remote sensing data is widely used in many fields such as environmental monitoring, urban planning, disaster warning, etc. However, night light remote sensing data is usually stored in HDF5 (Hierarchical Data Format version 5, H5) format, which has great advantages in data storage and transmission, but in practical applications it is often necessary to convert it to GeoTiff format for further analysis and processing.

[0003] At present, the main conversion technologies include manual conversion using remote sensing software (such as ENVI) and automatic conversion using remote sensing software scripts (such as ENVI-IDL). Although manual conversion is simple to operate, it has low processing efficiency and is difficult to cope with the conversion needs of large amounts of data. It also does not fully support variables, especially in terms of geometric data. Although remote sensing software scripts can achieve a certain degree of batch processing, they require high programming skills from users. In addition, remote sensing software scripts need to build a geometric lookup table for coordinate correction during the conversion process, which reduces the efficiency of data conversion.

[0004] It can be seen that the existing technology has problems such as low efficiency, low automation and incomplete support for variables when facing large-scale Suomi NPP night light remote sensing data conversion. In order to solve these problems, the present invention proposes a method for batch converting Suomi NPP night light remote sensing data from H5 format to GeoTiff format to improve data conversion efficiency and meet the needs of large-scale data processing. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings existing in the prior art, and propose a batch conversion method of Suomi NPP night light remote sensing data H5 to GeoTiff format, which can effectively solve the problems of low data conversion efficiency, low degree of automation and incomplete support for variables existing in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A batch conversion method of Suomi NPP night light remote sensing data H5 to GeoTiff format, comprising the following steps:

[0008] Step 1: Get H5 file names in batches

[0009] First, define the H5 file name list h5_files; then use the os.listdir() function to get all the files in the input folder in_dir, recorded as in_files; then traverse in_files to determine whether the extension of each file name ends with h5. If so, combine the input folder path with the file name and append it to the list h5_files;

[0010] Step 2: Set conversion parameters

[0011] Set the folder path for GeoTiff data output, recorded as out_dir; set the data scaling factor, recorded as scalar; set the invalid value, recorded as nodata;

[0012] Step 3: Read the H5 file structure

[0013] First, traverse each file in the list h5_files, recorded as h5_file. Use the h5py.File() function to read the data structure in the h5_file file, and obtain the level 1 group and level 2 group, recorded as groups_l1 and groups_l2 respectively; groups_l1 usually includes the options: 'All_Data' and 'Data_Products'; groups_l2 must include the options: 'VIIRS-DNB-GEO_All' and 'VIIRS-DNB-SDR_All'; based on groups_l2, obtain the geometric information items and sensor data record items of the level 3 group, recorded as g3_geo and g3_sdr respectively;

[0014] Step 4: Read the specified variable data

[0015] First, set the variable name of the dataset to be extracted, recorded as dataset_name, and define the dataset variable, recorded as dataset; then determine the variable type of dataset_name. If it is sdr type, extract the dataset in the 3-level group g3_sdr and multiply the dataset by the scaling factor scalar; if the variable type of dataset_name is geo, directly extract the dataset in the 3-level group g3_geo;

[0016] Step 5: Geographic coordinate correction;

[0017] Step 6: Export GeoTiff data.

[0018] Preferably, in step five, the following steps are specifically included:

[0019] (1) Define the coordinate validity check function chk_cds(), which is used to determine whether the input coordinates (lon, lat) are within the valid range {(lon, lat)|lon∈[-180, 180], lat∈[-90, 90]}. If yes, it returns True, otherwise it returns False;

[0020] (2) Read the initial geographic coordinate data, obtain the longitude coordinate matrix Lon and the latitude coordinate matrix Lat through the 3-level group g3_geo, calculate the number of Lon and Lat respectively, and record them as N_Lat and N_Lon. Let the variable rows = N_Lat, and the variable cols = N_Lon;

[0021] (3) Filter valid coordinate values ​​and define valid coordinate value constraints: valid_indices = (Lon>=-180) & (Lat>=-90) & (Lon<=180) & (Lat<=90); extract valid coordinate values ​​in Lon and Lat through valid_indices, and record them as Lonv and Latv respectively;

[0022] (4) Calculate the coordinate corner points after geometric correction. In the valid coordinate lists Lonv and Latv, use the minimum value min() and maximum value max() functions to calculate the coordinates of the upper left corner and lower right corner of the geometrically corrected image, which are recorded as: (LonMin, LatMax) and (LonMax, LatMin) respectively;

[0023] (5) Calculate the resolution. Based on the previous step, calculate the pixel resolution in the longitude and latitude directions, which are recorded as Lon_Res = (LonMax-LonMin) / (N_Lon*1.0) and Lat_Res = (LatMax-LatMin) / (N_Lat*1.0) respectively.

[0024] (6) Geometric correction: First, define a variable for storing the corrected data, denoted as data_array. This variable is a two-dimensional matrix of rows*cols, and the initial pixel value is set to nodata.

[0025] Then, we traverse the dataset variable dataset according to the rows and columns, and extract individual data one by one; let the pixel value of the i-th row and j-th column be recorded as _data=dataset[i,j], and extract the corresponding coordinate values ​​in the coordinate variables Lon and Lat, which are recorded as _lon=Lon[i,j] and _lat=Lat[i,j] respectively;

[0026] Then, the chk_cds() function is used to determine whether the coordinates (_lon, _lat) are valid; if valid, the row and column numbers of the coordinates (_lon, _lat) in the corrected data set data_array are calculated respectively; then: col = int((_lon-LonMin) / Lon_Res) and row = int((LatMax-_lat) / Lat_Res); further, to ensure that col and row are valid, coordinate boundary constraints are required, then: col = min(col, cols-1) and row = min(row, rows-1);

[0027] Finally, assign the pixel [row, col] value in data_array to _data.

[0028] Preferably, in step six, the following steps are specifically included:

[0029] First, set the output data file name, recorded as tif_name. The naming method of tif_name is: tif_name = out_dir + os.sep + os.path.basename (h5_file) [0: -3] + '_' + dataset_name + '.tif'.;

[0030] Then, use the gdal.GetDriverByName('GTiff').Create() function to create the output raster dataset according to the parameters tif_name, cols, and rows, and record it as dataset_out;

[0031] Then, use the SetGeoTransform() function to define the geographic spatial reference information for dataset_out according to the parameters (LonMin, Lon_Res, 0, LatMax, 0, -Lat_Res);

[0032] Next, use the SetProjection() function to define the projection coordinate system for dataset_out;

[0033] Finally, the WriteArray function is used to write the geometrically corrected dataset data_array into the raster band of dataset_out.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. In terms of data conversion quality, the present invention can ensure that the quality of the converted data meets the standards and will not increase data noise.

[0036] 2. In terms of batch data conversion, the present invention can automatically convert all H5 files in a specified folder. Compared with the manual conversion mode using ENVI software, the present invention can significantly improve the conversion efficiency.

[0037] 3. In terms of data conversion stability, the present invention can not only convert the sensor data record (SDR) of Suomi NPP night light remote sensing data, but also convert the geometric data (Geo) with stability. Compared with using ENVI software that can only perform SDR conversion, the present invention has significant advantages in geometric data conversion.

[0038] 4. In terms of data conversion time efficiency, the present invention takes about 3.92 minutes to convert a single data with a pixel size of 4064*3072, which is significantly faster than the ENVI IDL method (which takes at least 30 minutes due to the construction of a geometric lookup table). Therefore, the present invention has a higher time conversion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the present invention;

[0040] Figure 2 This is a diagram of the H5 file structure read in an embodiment of the present invention;

[0041] Figure 3 This is a data conversion effect diagram in an embodiment of the present invention;

[0042] Figure 4 This is a data conversion time analysis diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the advantages and features of the present invention, thereby making a clearer definition of the protection scope of the present invention. The embodiments described in the present invention are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, a batch conversion method of Suomi NPP night light remote sensing data H5 to GeoTiff format includes the following steps:

[0045] Step 1: Get H5 file names in batches

[0046] First, define the H5 file name list h5_files; then use the os.listdir() function to get all the files in the input folder in_dir, recorded as in_files; then, traverse in_files to determine whether the extension of each file name ends with h5. If so, combine the input folder path with the file name and append it to the list h5_files; the relevant code for this step is:

[0047] h5_files = [] #Define the H5 file name list

[0048]

[0049] Step 2: Set conversion parameters

[0050] Set the folder path for GeoTiff data output, recorded as out_dir; set the data scaling factor, recorded as scalar (if the variable is Radiance, scalar needs to be set to 1.0*10**9); set the invalid value, recorded as nodata=-999.3;

[0051] Step 3: Read the H5 file structure

[0052] First, traverse each file in the list h5_files, recorded as h5_file. Use the h5py.File() function to read the data structure in the h5_file file, and obtain the level 1 group and level 2 group, recorded as groups_l1 and groups_l2; groups_l1 usually includes the options: 'All_Data' and 'Data_Products'; groups_l2 must include the options: 'VIIRS-DNB-GEO_All' and 'VIIRS-DNB-SDR_All'; based on groups_l2, obtain the geometric information items and sensor data record items of the level 3 group, recorded as g3_geo and g3_sdr respectively. The relevant code for this step is:

[0053]

[0054] After this step, the structure of the read H5 file is as follows Figure 2 As shown in b.

[0055] Step 4: Read the specified variable data

[0056] First, set the variable name of the dataset to be extracted, recorded as dataset_name, and define the dataset variable, recorded as dataset; then determine the variable type of dataset_name. If it is sdr type, extract the dataset in the 3-level group g3_sdr and multiply the dataset by the scaling factor scalar; if the variable type of dataset_name is geo, directly extract the dataset in the 3-level group g3_geo. The relevant code for this step is:

[0057]

[0058] dataset=g3_sdr[dataset_name][:]

[0059] dataset = [[element*scalar for element in row]for row in dataset] # The dataset is uniformly multiplied by the scaling factor

[0060] dataset = np.array(dataset) # Convert to Numpy matrix

[0061] else:#geo: geometry information

[0062] dataset=g3_geo[dataset_name][:]

[0063] Step 5: Geographic coordinate correction;

[0064] (1) Define the coordinate validity check function chk_cds(), which is used to determine whether the input coordinates (lon, lat) are within the valid range {(lon, lat)|lon∈[-180, 180], lat∈[-90, 90]}. If yes, it returns True, otherwise it returns False;

[0065] (2) Read the initial geographic coordinate data, obtain the longitude coordinate matrix Lon and the latitude coordinate matrix Lat through the 3-level group g3_geo, calculate the number of Lon and Lat respectively, and record them as N_Lat and N_Lon. Let the variable rows = N_Lat, and the variable cols = N_Lon;

[0066] (3) Filter valid coordinate values ​​and define valid coordinate value constraints: valid_indices = (Lon>=-180) & (Lat>=-90) & (Lon<=180) & (Lat<=90); extract valid coordinate values ​​in Lon and Lat through valid_indices, and record them as Lonv and Latv respectively;

[0067] (4) Calculate the coordinate corner points after geometric correction. In the valid coordinate lists Lonv and Latv, use the minimum value min() and maximum value max() functions to calculate the coordinates of the upper left corner and lower right corner of the geometrically corrected image, which are recorded as: (LonMin, LatMax) and (LonMax, LatMin) respectively;

[0068] (5) Calculate the resolution. Based on the previous step, calculate the pixel resolution in the longitude and latitude directions, which are recorded as Lon_Res = (LonMax-LonMin) / (N_Lon*1.0) and Lat_Res = (LatMax-LatMin) / (N_Lat*1.0) respectively.

[0069] (6) Geometric correction: First, define a variable for storing the corrected data, denoted as data_array. This variable is a two-dimensional matrix of rows*cols, and the initial pixel value is set to nodata.

[0070] Then, we traverse the dataset variable dataset according to the rows and columns, and extract individual data one by one; let the pixel value of the i-th row and j-th column be recorded as _data=dataset[i,j], and extract the corresponding coordinate values ​​in the coordinate variables Lon and Lat, which are recorded as _lon=Lon[i,j] and _lat=Lat[i,j] respectively;

[0071] Then, the chk_cds() function is used to determine whether the coordinates (_lon, _lat) are valid; if valid, the row and column numbers of the coordinates (_lon, _lat) in the corrected data set data_array are calculated respectively; then: col = int((_lon-LonMin) / Lon_Res) and row = int((LatMax-_lat) / Lat_Res); further, to ensure that col and row are valid, coordinate boundary constraints are required, then: col = min(col, cols-1) and row = min(row, rows-1);

[0072] Finally, assign the pixel [row, col] value in data_array to _data. The relevant code for this step is:

[0073]

[0074]

[0075] Step 6: Export GeoTiff data

[0076] First, set the output data file name, recorded as tif_name. The naming method of tif_name is: tif_name = out_dir + os.sep + os.path.basename (h5_file) [0: -3] + '_' + dataset_name + '.tif'.;

[0077] Then, use the gdal.GetDriverByName('GTiff').Create() function to create the output raster dataset according to the parameters tif_name, cols, and rows, and record it as dataset_out;

[0078] Then, use the SetGeoTransform() function to define the geographic spatial reference information for dataset_out according to the parameters (LonMin, Lon_Res, 0, LatMax, 0, -Lat_Res);

[0079] Next, use the SetProjection() function to define the projection coordinate system for dataset_out;

[0080] Finally, the WriteArray function is used to write the geometrically corrected dataset data_array into the raster band of dataset_out.

[0081] The relevant code for this step is:

[0082]

[0083] band.FlushCache()

[0084] After the above steps, Suomi NPP night light remote sensing data can be quickly and batch converted from H5 to GeoTiff format. The final converted geometry data sample (variable name set to SatelliteZenithAngle) and sensor data record sample (variable name set to Radiance) are as follows: Figure 3 b. The average time for a single data conversion is 3.92 minutes. Figure 4 shown.

[0085] In summary, the present invention can improve data conversion efficiency and meet the needs of large-scale data processing.

[0086] The description and practice disclosed in the present invention are easy to think and understand for ordinary technicians in the technical field, and several improvements and modifications can be made without departing from the principles of the present invention. Therefore, modifications or improvements made without departing from the spirit of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A batch conversion method of Suomi NPP night light remote sensing data H5 to GeoTiff format, characterized in that: The following steps are involved: Step 1: Get H5 file names in batches First, define the H5 file name list h5_files; then use the os.listdir() function to get all the files in the input folder in_dir, recorded as in_files; then traverse in_files to determine whether the extension of each file name ends with h5. If so, combine the input folder path with the file name and append it to the list h5_files; Step 2: Set conversion parameters Set the folder path for GeoTiff data output, recorded as out_dir; set the data scaling factor, recorded as scalar; set the invalid value, recorded as nodata; Step 3: Read the H5 file structure First, traverse each file in the list h5_files, recorded as h5_file. Use the h5py.File() function to read the data structure in the h5_file file, and obtain the level 1 group and level 2 group, recorded as groups_l1 and groups_l2 respectively; groups_l1 includes the options: 'All_Data' and 'Data_Products'; groups_l2 must include the options: 'VIIRS-DNB-GEO_All' and 'VIIRS-DNB-SDR_All'; based on groups_l2, obtain the geometric information items and sensor data record items of the level 3 group, recorded as g3_geo and g3_sdr respectively; Step 4: Read the specified variable data First, set the variable name of the dataset to be extracted, recorded as dataset_name, and define the dataset variable, recorded as dataset; then determine the variable type of dataset_name. If it is sdr type, extract the dataset in the 3-level group g3_sdr and multiply the dataset by the scaling factor scalar; if the variable type of dataset_name is geo, directly extract the dataset in the 3-level group g3_geo; Step 5: Geographic coordinate correction; Step 6: Export GeoTiff data.

2. The method for batch conversion of Suomi NPP night light remote sensing data H5 to GeoTiff format according to claim 1, characterized in that: In step five, the following steps are specifically included: (1) Define the coordinate validity check function chk_cds(), which is used to determine whether the input coordinates (lon, lat) are within the valid range {(lon, lat)|lon∈[-180, 180], lat∈[-90, 90]}. If yes, it returns True, otherwise it returns False; (2) Read the initial geographic coordinate data, obtain the longitude coordinate matrix Lon and the latitude coordinate matrix Lat through the level 3 group g3_geo, calculate the number of Lon and Lat respectively, record them as N_Lat and N_Lon. Let the variable rows = N_Lat, the variable cols = N_Lon; (3) Filter valid coordinate values ​​and define valid coordinate value constraints: valid_indices = (Lon>=-180) & (Lat>=-90) & (Lon<=180) & (Lat<=90); extract valid coordinate values ​​in Lon and Lat through valid_indices, and record them as Lonv and Latv respectively; (4) Calculate the coordinate corner points after geometric correction. In the valid coordinate lists Lonv and Latv, use the minimum value min() and maximum value max() functions to calculate the coordinates of the upper left corner and lower right corner of the geometrically corrected image, which are recorded as: (LonMin, LatMax) and (LonMax, LatMin) respectively; (5) Calculate the resolution. Based on the previous step, calculate the pixel resolution in the longitude and latitude directions, which are recorded as Lon_Res = (LonMax-LonMin) / (N_Lon*1.0) and Lat_Res = (LatMax-LatMin) / (N_Lat*1.0) respectively. (6) Geometric correction: First, define a variable for storing the corrected data, denoted as data_array. This variable is a two-dimensional matrix of rows*cols, and the initial pixel value is set to nodata. Then, we traverse the dataset variable dataset according to the rows and columns, and extract individual data one by one; let the pixel value of the i-th row and j-th column be recorded as _data=dataset[i,j], and extract the corresponding coordinate values ​​in the coordinate variables Lon and Lat, which are recorded as _lon=Lon[i,j] and _lat=Lat[i,j] respectively; Then, the chk_cds() function is used to determine whether the coordinates (_lon, _lat) are valid; if valid, the row and column numbers of the coordinates (_lon, _lat) in the corrected data set data_array are calculated respectively; then: col = int((_lon-LonMin) / Lon_Res) and row = int((LatMax-_lat) / Lat_Res); further, to ensure that col and row are valid, coordinate boundary constraints are required, then: col = min(col, cols-1) and row = min(row, rows-1); Finally, assign the pixel [row, col] value in data_array to _data.

3. The method for batch conversion of Suomi NPP night light remote sensing data H5 to GeoTiff format according to claim 1, characterized in that: In step six, the following steps are specifically included: First, set the output data file name, recorded as tif_name. The naming method of tif_name is: tif_name = out_dir + os.sep + os.path.basename (h5_file) [0: -3] + '_' + dataset_name + '.tif'.; Then, use the gdal.GetDriverByName('GTiff').Create() function to create the output raster dataset according to the parameters tif_name, cols, and rows, and record it as dataset_out; Then, use the SetGeoTransform() function to define the geographic spatial reference information for dataset_out according to the parameters (LonMin, Lon_Res, 0, LatMax, 0, -Lat_Res); Next, use the SetProjection() function to define the projection coordinate system for dataset_out; Finally, the WriteArray function is used to write the geometrically corrected dataset data_array into the raster band of dataset_out.

Images